2024-01-27T03:20:48.035 INFO:root:teuthology version: 0.0.1.dev251+gc856e0f 2024-01-27T03:20:48.035 DEBUG:teuthology.run:Teuthology command: teuthology --name lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi --owner scheduled_lflores@teuthology --verbose --archive /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300 --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/orig.config.yaml 2024-01-27T03:20:48.073 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-27T03:20:48.155 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300 branch: wip-read-balancer-mgr-module description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7535300' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-read-balancer-mgr-module ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 5ffe0641f18e6ec692535294010d9456630e7923 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5ffe0641f18e6ec692535294010d9456630e7923 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-read-balancer-mgr-module sha1: 5ffe0641f18e6ec692535294010d9456630e7923 owner: scheduled_lflores@teuthology priority: 74 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 877 sha1: 5ffe0641f18e6ec692535294010d9456630e7923 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-read-balancer-mgr-module suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5ffe0641f18e6ec692535294010d9456630e7923 targets: smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE7jbvzpRRcKWxC8MKwEp6/2Eg7/ebLoAtLx3fCHrCP4j6YgaKbk2gzjCrCnqBSa8R7LP8NJKUc2BF0mFs8jIH8= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJpZBxOlpfYmIDdeeZrTsodR/7nDNca1GqxhvsdenHo1pug6H89ZquNR7aY5aBc19vkT9OGARRG+poLeF0c8MI4= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c856e0f9244dd29a50697123b480d63f16a4ed81 timestamp: 2024-01-27_03:09:32 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-27T03:20:48.155 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa; will attempt to use it 2024-01-27T03:20:48.156 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa/tasks 2024-01-27T03:20:48.156 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-27T03:20:48.158 INFO:teuthology.task.internal:Checking packages... 2024-01-27T03:20:48.180 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5ffe0641f18e6ec692535294010d9456630e7923' 2024-01-27T03:20:48.180 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-27T03:20:48.180 INFO:teuthology.packaging:ref: None 2024-01-27T03:20:48.180 INFO:teuthology.packaging:tag: None 2024-01-27T03:20:48.180 INFO:teuthology.packaging:branch: wip-read-balancer-mgr-module 2024-01-27T03:20:48.180 INFO:teuthology.packaging:sha1: 5ffe0641f18e6ec692535294010d9456630e7923 2024-01-27T03:20:48.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-read-balancer-mgr-module 2024-01-27T03:20:48.323 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-926.g5ffe0641 2024-01-27T03:20:48.325 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-27T03:20:48.331 INFO:teuthology.task.internal:no buildpackages task found 2024-01-27T03:20:48.331 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-27T03:20:48.379 INFO:teuthology.task.internal:Saving configuration 2024-01-27T03:20:48.394 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-27T03:20:48.399 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-27T03:20:48.425 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-27 03:15:43.449956', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE7jbvzpRRcKWxC8MKwEp6/2Eg7/ebLoAtLx3fCHrCP4j6YgaKbk2gzjCrCnqBSa8R7LP8NJKUc2BF0mFs8jIH8='} 2024-01-27T03:20:48.445 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-27 03:15:43.451957', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJpZBxOlpfYmIDdeeZrTsodR/7nDNca1GqxhvsdenHo1pug6H89ZquNR7aY5aBc19vkT9OGARRG+poLeF0c8MI4='} 2024-01-27T03:20:48.445 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-27T03:20:48.451 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-01-27T03:20:48.452 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-01-27T03:20:48.452 INFO:teuthology.run_tasks:Running task console_log... 2024-01-27T03:20:48.533 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4891faddc0>, signals=[15]) 2024-01-27T03:20:48.534 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-27T03:20:48.539 INFO:teuthology.task.internal:Opening connections... 2024-01-27T03:20:48.540 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T03:20:48.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:20:48.633 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T03:20:48.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:20:48.720 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-27T03:20:48.727 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-01-27T03:20:48.744 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-01-27T03:20:48.744 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-01-27T03:20:48.799 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-27T03:20:48.800 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-01-27T03:20:48.800 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-27T03:20:48.800 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-27T03:20:48.800 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-27T03:20:48.800 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-01-27T03:20:48.820 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-01-27T03:20:48.836 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-01-27T03:20:48.837 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-01-27T03:20:48.892 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-01-27T03:20:48.892 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-01-27T03:20:48.892 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-01-27T03:20:48.892 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-01-27T03:20:48.892 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-01-27T03:20:48.892 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-01-27T03:20:48.892 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-27T03:20:48.892 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-01-27T03:20:48.893 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-01-27T03:20:48.893 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-27T03:20:48.893 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-01-27T03:20:48.893 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-27T03:20:48.893 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-27T03:20:48.893 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-27T03:20:48.893 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-01-27T03:20:48.911 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-27T03:20:48.920 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-27T03:20:48.934 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-27T03:20:48.935 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-01-27T03:20:48.937 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-01-27T03:20:48.952 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-27T03:20:48.957 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-27T03:20:48.958 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-01-27T03:20:48.992 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-01-27T03:20:49.023 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-27T03:20:49.078 INFO:teuthology.run_tasks:Running task kernel... 2024-01-27T03:20:49.092 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-27T03:20:49.092 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-27T03:20:49.092 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-27T03:20:49.093 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-27T03:20:49.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T03:20:49.108 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-01-27T03:20:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T03:20:49.109 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-01-27T03:20:49.162 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-361.el9.x86_64 2024-01-27T03:20:49.162 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-361.el9.x86_64 2024-01-27T03:20:49.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-01-27T03:20:49.164 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-361.el9.x86_64 2024-01-27T03:20:49.164 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-361.el9.x86_64 2024-01-27T03:20:49.164 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-01-27T03:20:51.171 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2024-01-27T03:20:51.376 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2024-01-27T03:20:52.701 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-01-27T03:20:52.813 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-01-27T03:20:53.433 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-01-27T03:20:54.320 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.0 MB 00:01 2024-01-27T03:20:55.185 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-01-27T03:20:57.215 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-01-27T03:20:59.931 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 5.9 MB 00:01 2024-01-27T03:21:02.147 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.9 MB 00:01 2024-01-27T03:21:02.153 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 15 kB 00:01 2024-01-27T03:21:03.072 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2024-01-27T03:21:03.747 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-01-27T03:21:07.969 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-01-27T03:21:10.268 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-27T03:21:10.268 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-27T03:21:10.361 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:21:10.373 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:21:10.373 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-411.el9 CentOS-BaseOS 6.0 M 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 5.14.0-411.el9 CentOS-BaseOS 21 M 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 5.14.0-411.el9 CentOS-BaseOS 39 M 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core x86_64 5.14.0-411.el9 CentOS-BaseOS 33 M 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 99 M 2024-01-27T03:21:10.374 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 122 M 2024-01-27T03:21:10.375 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:21:11.037 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): kernel-5.14.0-411.el9.x86_64.rpm 9.1 MB/s | 6.0 MB 00:00 2024-01-27T03:21:11.512 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): kernel-core-5.14.0-411.el9.x86_64.rpm 18 MB/s | 21 MB 00:01 2024-01-27T03:21:12.196 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): kernel-modules-5.14.0-411.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-01-27T03:21:13.296 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): kernel-modules-core-5.14.0-411.el9.x86_6 15 MB/s | 33 MB 00:02 2024-01-27T03:21:13.299 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:21:13.299 INFO:teuthology.orchestra.run.smithi162.stdout:Total 34 MB/s | 99 MB 00:02 2024-01-27T03:21:13.299 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:21:13.671 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:21:13.671 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:21:14.703 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:21:14.703 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:21:17.126 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:21:17.825 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-core-5.14.0-411.el9.x86_64 1/4 2024-01-27T03:21:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-5.14.0-411.el9.x86_64 2/4 2024-01-27T03:21:19.609 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 20 MB 00:15 2024-01-27T03:21:20.704 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-411.el9.x86_64 2/4 2024-01-27T03:21:20.887 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-5.14.0-411.el9.x86_64 3/4 2024-01-27T03:21:24.364 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-01-27T03:21:25.592 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-411.el9.x86_64 3/4 2024-01-27T03:21:25.715 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:21:26.587 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-27T03:21:26.588 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-27T03:21:26.697 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:21:26.703 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-411.el9 CentOS-BaseOS 6.0 M 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 5.14.0-411.el9 CentOS-BaseOS 21 M 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 5.14.0-411.el9 CentOS-BaseOS 39 M 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core x86_64 5.14.0-411.el9 CentOS-BaseOS 33 M 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-01-27T03:21:26.704 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:21:26.705 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 99 M 2024-01-27T03:21:26.705 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 122 M 2024-01-27T03:21:26.705 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:21:27.040 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): kernel-5.14.0-411.el9.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-01-27T03:21:27.574 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): kernel-core-5.14.0-411.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-01-27T03:21:28.149 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): kernel-modules-core-5.14.0-411.el9.x86_6 30 MB/s | 33 MB 00:01 2024-01-27T03:21:29.241 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): kernel-modules-5.14.0-411.el9.x86_64.rpm 15 MB/s | 39 MB 00:02 2024-01-27T03:21:29.244 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:21:29.244 INFO:teuthology.orchestra.run.smithi110.stdout:Total 39 MB/s | 99 MB 00:02 2024-01-27T03:21:29.244 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:21:29.621 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:21:29.622 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:21:30.430 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-core-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:21:30.665 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:21:30.666 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:21:33.063 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:21:33.728 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-core-5.14.0-411.el9.x86_64 1/4 2024-01-27T03:21:33.860 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-5.14.0-411.el9.x86_64 2/4 2024-01-27T03:21:36.633 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-411.el9.x86_64 2/4 2024-01-27T03:21:36.813 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-5.14.0-411.el9.x86_64 3/4 2024-01-27T03:21:41.515 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-411.el9.x86_64 3/4 2024-01-27T03:21:41.741 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:21:46.463 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-core-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:22:00.071 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:22:00.071 INFO:teuthology.orchestra.run.smithi162.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-27T03:22:00.071 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:22:00.157 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:22:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:22:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-411.el9.x86_64 1/4 2024-01-27T03:22:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-5.14.0-411.el9.x86_64 2/4 2024-01-27T03:22:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-5.14.0-411.el9.x86_64 3/4 2024-01-27T03:22:00.985 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-core-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:22:00.985 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:22:00.985 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:22:00.985 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-411.el9.x86_64 2024-01-27T03:22:00.985 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core-5.14.0-411.el9.x86_64 2024-01-27T03:22:00.986 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-5.14.0-411.el9.x86_64 2024-01-27T03:22:00.986 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core-5.14.0-411.el9.x86_64 2024-01-27T03:22:00.986 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:22:00.986 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:22:01.493 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-01-27T03:22:01.663 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-411.el9.x86_64 2024-01-27T03:22:01.663 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-411.el9.x86_64 2024-01-27T03:22:01.663 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-411.el9.x86_64 2024-01-27T03:22:01.664 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-27T03:22:01.664 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-27T03:22:01.664 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-01-27T03:22:01.664 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-01-27T03:22:01.664 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-01-27T03:22:01.772 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-411.el9.x86_64 2024-01-27T03:22:01.773 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-411.el9.x86_64 2024-01-27T03:22:01.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.06 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 68.el9 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:Group : Unspecified 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 8264254 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-01-27T03:22:01.816 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : builder@centos.org 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:22:01.817 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-01-27T03:22:01.818 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-27T03:22:01.818 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 5.14.0-411.el9.x86_64 2024-01-27T03:22:01.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-27T03:22:02.410 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-01-27T03:22:03.781 INFO:teuthology.orchestra.run.smithi162.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-27T03:22:03.820 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-01-27T03:22:03.822 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-01-27T03:22:03.838 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.TjpM8IOa1e 2024-01-27T03:22:03.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TjpM8IOa1e 2024-01-27T03:22:03.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.TjpM8IOa1e 2024-01-27T03:22:04.046 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.TjpM8IOa1e is 6KB 2024-01-27T03:22:04.094 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.TjpM8IOa1e 2024-01-27T03:22:04.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-01-27T03:22:04.172 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-27T03:22:04.172 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-27T03:22:04.172 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-27T03:22:04.172 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-411.el9.x86_64.conf 2024-01-27T03:22:04.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-411.el9.x86_64 2024-01-27T03:22:04.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-01-27T03:22:15.494 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:22:15.494 INFO:teuthology.orchestra.run.smithi110.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-27T03:22:15.494 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:22:15.562 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:22:15.980 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:22:15.980 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-411.el9.x86_64 1/4 2024-01-27T03:22:15.980 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-5.14.0-411.el9.x86_64 2/4 2024-01-27T03:22:15.980 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-5.14.0-411.el9.x86_64 3/4 2024-01-27T03:22:16.396 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-core-5.14.0-411.el9.x86_64 4/4 2024-01-27T03:22:16.396 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:22:16.397 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:22:16.397 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-411.el9.x86_64 2024-01-27T03:22:16.397 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core-5.14.0-411.el9.x86_64 2024-01-27T03:22:16.397 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-5.14.0-411.el9.x86_64 2024-01-27T03:22:16.397 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core-5.14.0-411.el9.x86_64 2024-01-27T03:22:16.397 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:22:16.397 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:22:16.887 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-01-27T03:22:17.027 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-411.el9.x86_64 2024-01-27T03:22:17.027 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-411.el9.x86_64 2024-01-27T03:22:17.027 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-411.el9.x86_64 2024-01-27T03:22:17.027 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-27T03:22:17.027 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-27T03:22:17.027 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-27T03:22:17.028 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-01-27T03:22:17.028 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-01-27T03:22:17.136 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-411.el9.x86_64 2024-01-27T03:22:17.136 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-411.el9.x86_64 2024-01-27T03:22:17.136 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2024-01-27T03:22:17.178 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.06 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 68.el9 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Group : Unspecified 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 8264254 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-27T03:22:17.179 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : builder@centos.org 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:22:17.180 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2024-01-27T03:22:17.181 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-27T03:22:17.181 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 5.14.0-411.el9.x86_64 2024-01-27T03:22:17.181 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-27T03:22:17.786 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-01-27T03:22:19.163 INFO:teuthology.orchestra.run.smithi110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-27T03:22:19.205 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-01-27T03:22:19.207 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-01-27T03:22:19.223 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.1M4B21lhxG 2024-01-27T03:22:19.224 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1M4B21lhxG 2024-01-27T03:22:19.293 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.1M4B21lhxG 2024-01-27T03:22:19.434 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.1M4B21lhxG is 6KB 2024-01-27T03:22:19.482 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.1M4B21lhxG 2024-01-27T03:22:19.496 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2024-01-27T03:22:19.561 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-27T03:22:19.561 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-27T03:22:19.561 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-27T03:22:19.561 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-411.el9.x86_64.conf 2024-01-27T03:22:19.563 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-411.el9.x86_64 2024-01-27T03:22:19.712 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-01-27T03:22:34.340 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-27T03:22:34.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-01-27T03:22:34.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:22:49.726 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-27T03:22:49.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-27T03:22:49.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:22:52.860 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-01-27T03:23:01.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-01-27T03:23:01.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:23:04.928 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-01-27T03:23:08.220 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-01-27T03:23:16.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-01-27T03:23:16.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:23:17.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-27T03:23:17.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:23:35.360 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-01-27T03:23:35.616 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-01-27T03:23:47.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-27T03:23:47.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:23:50.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-01-27T03:23:50.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:23:50.684 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-01-27T03:23:50.733 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-01-27T03:23:51.142 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-01-27T03:23:51.142 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-411.el9.x86_64"... 2024-01-27T03:23:51.142 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-01-27T03:23:51.157 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-411.el9.x86_64 2024-01-27T03:23:51.157 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-411.el9.x86_64 vs 5.14.0-411.el9.x86_64 2024-01-27T03:23:51.157 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-27T03:23:51.157 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-27T03:23:52.158 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-27T03:23:52.159 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-27T03:23:52.253 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-01-27T03:23:52.302 DEBUG:teuthology.parallel:result is None 2024-01-27T03:24:05.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-27T03:24:05.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:24:05.954 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-01-27T03:24:06.345 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-27T03:24:06.346 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-411.el9.x86_64"... 2024-01-27T03:24:06.346 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-01-27T03:24:06.361 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-411.el9.x86_64 2024-01-27T03:24:06.362 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-411.el9.x86_64 vs 5.14.0-411.el9.x86_64 2024-01-27T03:24:06.362 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-27T03:24:06.362 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-27T03:24:07.362 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-27T03:24:07.362 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-27T03:24:07.461 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-01-27T03:24:07.498 DEBUG:teuthology.parallel:result is None 2024-01-27T03:24:07.499 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-27T03:24:07.514 INFO:teuthology.task.internal:Creating test directory... 2024-01-27T03:24:07.514 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-27T03:24:07.517 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-27T03:24:07.536 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-27T03:24:07.545 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-27T03:24:07.576 INFO:teuthology.task.internal:Creating archive directory... 2024-01-27T03:24:07.576 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-27T03:24:07.579 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-27T03:24:07.679 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-27T03:24:07.686 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-27T03:24:07.686 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-27T03:24:07.689 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-27T03:24:07.722 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-27T03:24:07.723 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-27T03:24:07.736 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-27T03:24:07.737 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-27T03:24:07.739 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-27T03:24:07.747 INFO:teuthology.task.internal:Configuring sudo... 2024-01-27T03:24:07.747 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-27T03:24:07.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-27T03:24:07.811 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-27T03:24:07.820 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-27T03:24:07.820 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-27T03:24:07.851 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-27T03:24:07.867 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-27T03:24:07.947 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-27T03:24:08.001 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-27T03:24:08.081 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-27T03:24:08.126 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:24:08.126 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-27T03:24:08.193 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-27T03:24:08.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-27T03:24:08.294 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-27T03:24:08.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-27T03:24:08.422 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:24:08.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-27T03:24:08.490 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-01-27T03:24:08.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-01-27T03:24:08.533 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T03:24:08.594 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T03:24:09.093 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-27T03:24:09.101 INFO:teuthology.task.internal:Starting timer... 2024-01-27T03:24:09.101 INFO:teuthology.run_tasks:Running task pcp... 2024-01-27T03:24:09.124 INFO:teuthology.run_tasks:Running task selinux... 2024-01-27T03:24:09.157 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-01-27T03:24:09.213 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-01-27T03:24:09.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-01-27T03:24:09.277 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-01-27T03:24:09.279 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-27T03:24:09.279 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-01-27T03:24:09.309 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-01-27T03:24:09.309 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-01-27T03:24:09.335 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-01-27T03:24:09.336 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-01-27T03:24:09.336 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T03:24:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T03:24:09.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T03:24:09.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T03:24:09.406 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-27T03:24:09.407 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-01-27T03:24:09.452 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-01-27T03:24:09.495 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-27T03:24:09.506 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-27T03:24:09.512 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-27T03:24:09.639 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-27T03:24:09.653 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-27T03:24:09.655 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi110.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-27T03:28:14.138 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-01-27T03:28:14.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-27T03:28:14.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:28:14.232 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-01-27T03:28:14.311 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-01-27T03:28:14.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-01-27T03:28:14.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T03:28:14.401 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-01-27T03:28:14.487 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-01-27T03:28:14.488 INFO:teuthology.run_tasks:Running task clock... 2024-01-27T03:28:14.498 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-27T03:28:14.498 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-27T03:28:14.499 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T03:28:14.501 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-27T03:28:14.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T03:28:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-27T03:28:14.546 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-27T03:28:14.559 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-27T03:28:14.576 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-27T03:28:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-01-27T03:28:14.589 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-01-27T03:28:14.604 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-01-27T03:28:14.605 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-27T03:28:14.618 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-01-27T03:28:14.622 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-27T03:28:14.634 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-27T03:28:14.650 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-27T03:28:14.677 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-01-27T03:28:14.679 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T03:28:14.679 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-01-27T03:28:14.679 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T03:28:14.680 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T03:28:14.680 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T03:28:14.680 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T03:28:14.702 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-01-27T03:28:14.706 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T03:28:14.706 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-01-27T03:28:14.706 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T03:28:14.706 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T03:28:14.706 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T03:28:14.706 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T03:28:14.706 INFO:teuthology.run_tasks:Running task install... 2024-01-27T03:28:14.714 DEBUG:teuthology.task.install:project ceph 2024-01-27T03:28:14.715 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5ffe0641f18e6ec692535294010d9456630e7923'}} 2024-01-27T03:28:14.715 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5ffe0641f18e6ec692535294010d9456630e7923'} 2024-01-27T03:28:14.715 INFO:teuthology.task.install:Using flavor: default 2024-01-27T03:28:14.720 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-01-27T03:28:14.720 INFO:teuthology.task.install:extra packages: [] 2024-01-27T03:28:14.721 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5ffe0641f18e6ec692535294010d9456630e7923', 'tag': None, 'wait_for_package': False} 2024-01-27T03:28:14.721 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ffe0641f18e6ec692535294010d9456630e7923 2024-01-27T03:28:14.723 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5ffe0641f18e6ec692535294010d9456630e7923', 'tag': None, 'wait_for_package': False} 2024-01-27T03:28:14.723 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ffe0641f18e6ec692535294010d9456630e7923 2024-01-27T03:28:14.867 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-read-balancer-mgr-module/5ffe0641f18e6ec692535294010d9456630e7923/centos/9/flavors/default/ 2024-01-27T03:28:14.867 INFO:teuthology.task.install.rpm:Package version is 19.0.0-926.g5ffe0641 2024-01-27T03:28:14.875 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-read-balancer-mgr-module/5ffe0641f18e6ec692535294010d9456630e7923/centos/9/flavors/default/ 2024-01-27T03:28:14.875 INFO:teuthology.task.install.rpm:Package version is 19.0.0-926.g5ffe0641 2024-01-27T03:28:15.065 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-read-balancer-mgr-module/5ffe0641f18e6ec692535294010d9456630e7923/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-read-balancer-mgr-module/5ffe0641f18e6ec692535294010d9456630e7923/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-read-balancer-mgr-module/5ffe0641f18e6ec692535294010d9456630e7923/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-27T03:28:15.065 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:28:15.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-27T03:28:15.069 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-read-balancer-mgr-module/5ffe0641f18e6ec692535294010d9456630e7923/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-read-balancer-mgr-module/5ffe0641f18e6ec692535294010d9456630e7923/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-read-balancer-mgr-module/5ffe0641f18e6ec692535294010d9456630e7923/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-27T03:28:15.069 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:28:15.069 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-27T03:28:15.093 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-01-27T03:28:15.094 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5ffe0641f18e6ec692535294010d9456630e7923/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-27T03:28:15.097 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-01-27T03:28:15.097 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5ffe0641f18e6ec692535294010d9456630e7923/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-27T03:28:15.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-27T03:28:15.165 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-27T03:28:15.242 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-27T03:28:15.244 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-27T03:28:15.308 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-01-27T03:28:15.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-01-27T03:28:15.311 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-01-27T03:28:15.312 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-01-27T03:28:15.545 INFO:teuthology.orchestra.run.smithi162.stdout:74 files removed 2024-01-27T03:28:15.555 INFO:teuthology.orchestra.run.smithi110.stdout:74 files removed 2024-01-27T03:28:15.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2024-01-27T03:28:15.582 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw 2024-01-27T03:28:16.592 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 114 kB/s | 80 kB 00:00 2024-01-27T03:28:16.601 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 114 kB/s | 80 kB 00:00 2024-01-27T03:28:17.318 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 14 kB/s | 9.5 kB 00:00 2024-01-27T03:28:17.385 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 13 kB/s | 9.5 kB 00:00 2024-01-27T03:28:17.952 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-01-27T03:28:17.969 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2024-01-27T03:28:18.730 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 8.8 MB/s | 6.4 MB 00:00 2024-01-27T03:28:18.754 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 8.3 MB/s | 6.4 MB 00:00 2024-01-27T03:28:20.246 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-01-27T03:28:20.506 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-01-27T03:28:21.972 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-01-27T03:28:22.288 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-01-27T03:28:24.014 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-01-27T03:28:27.828 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 19 MB 00:04 2024-01-27T03:28:27.898 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 15 MB/s | 5.9 MB 00:00 2024-01-27T03:28:29.716 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-01-27T03:28:30.587 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 28 MB/s | 20 MB 00:00 2024-01-27T03:28:31.929 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 10 MB/s | 5.9 MB 00:00 2024-01-27T03:28:33.746 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-01-27T03:28:35.512 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-01-27T03:28:36.342 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 20 MB 00:02 2024-01-27T03:28:38.147 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:28:38.156 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:28:38.156 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 9.7 M 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 3.4 M 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 3.1 M 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 5.1 M 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 20 M 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 25 k 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-27T03:28:38.157 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 743 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 498 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 5.1 M 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 45 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 130 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 161 k 2024-01-27T03:28:38.158 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 320 k 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 298 k 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 99 k 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout:Install 27 Packages 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 2 Packages 2024-01-27T03:28:38.159 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:28:38.160 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 57 M 2024-01-27T03:28:38.160 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:28:38.996 INFO:teuthology.orchestra.run.smithi110.stdout:(1/29): ceph-base-19.0.0-926.g5ffe0641.el9.x86_ 7.8 MB/s | 5.1 MB 00:00 2024-01-27T03:28:39.063 INFO:teuthology.orchestra.run.smithi110.stdout:(2/29): ceph-selinux-19.0.0-926.g5ffe0641.el9.x 378 kB/s | 25 kB 00:00 2024-01-27T03:28:39.155 INFO:teuthology.orchestra.run.smithi110.stdout:(3/29): libcephfs2-19.0.0-926.g5ffe0641.el9.x86 7.9 MB/s | 743 kB 00:00 2024-01-27T03:28:39.280 INFO:teuthology.orchestra.run.smithi110.stdout:(4/29): ceph-radosgw-19.0.0-926.g5ffe0641.el9.x 10 MB/s | 9.7 MB 00:00 2024-01-27T03:28:39.305 INFO:teuthology.orchestra.run.smithi110.stdout:(5/29): libradosstriper1-19.0.0-926.g5ffe0641.e 3.2 MB/s | 498 kB 00:00 2024-01-27T03:28:39.389 INFO:teuthology.orchestra.run.smithi110.stdout:(6/29): python3-ceph-argparse-19.0.0-926.g5ffe0 541 kB/s | 45 kB 00:00 2024-01-27T03:28:39.465 INFO:teuthology.orchestra.run.smithi110.stdout:(7/29): python3-ceph-common-19.0.0-926.g5ffe064 1.7 MB/s | 130 kB 00:00 2024-01-27T03:28:39.556 INFO:teuthology.orchestra.run.smithi110.stdout:(8/29): librgw2-19.0.0-926.g5ffe0641.el9.x86_64 19 MB/s | 5.1 MB 00:00 2024-01-27T03:28:39.581 INFO:teuthology.orchestra.run.smithi110.stdout:(9/29): python3-cephfs-19.0.0-926.g5ffe0641.el9 1.3 MB/s | 161 kB 00:00 2024-01-27T03:28:39.615 INFO:teuthology.orchestra.run.smithi110.stdout:(10/29): python3-rados-19.0.0-926.g5ffe0641.el9 5.3 MB/s | 320 kB 00:00 2024-01-27T03:28:39.640 INFO:teuthology.orchestra.run.smithi110.stdout:(11/29): python3-rbd-19.0.0-926.g5ffe0641.el9.x 5.0 MB/s | 298 kB 00:00 2024-01-27T03:28:39.665 INFO:teuthology.orchestra.run.smithi110.stdout:(12/29): python3-rgw-19.0.0-926.g5ffe0641.el9.x 1.9 MB/s | 99 kB 00:00 2024-01-27T03:28:39.782 INFO:teuthology.orchestra.run.smithi110.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-01-27T03:28:39.808 INFO:teuthology.orchestra.run.smithi110.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-01-27T03:28:39.841 INFO:teuthology.orchestra.run.smithi110.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 755 kB/s | 44 kB 00:00 2024-01-27T03:28:39.908 INFO:teuthology.orchestra.run.smithi110.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-01-27T03:28:39.933 INFO:teuthology.orchestra.run.smithi110.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-01-27T03:28:39.959 INFO:teuthology.orchestra.run.smithi110.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-01-27T03:28:40.501 INFO:teuthology.orchestra.run.smithi110.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 545 kB/s | 309 kB 00:00 2024-01-27T03:28:40.609 INFO:teuthology.orchestra.run.smithi110.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-01-27T03:28:40.718 INFO:teuthology.orchestra.run.smithi110.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-01-27T03:28:40.802 INFO:teuthology.orchestra.run.smithi110.stdout:(22/29): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-01-27T03:28:40.852 INFO:teuthology.orchestra.run.smithi110.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-01-27T03:28:40.911 INFO:teuthology.orchestra.run.smithi110.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-01-27T03:28:41.061 INFO:teuthology.orchestra.run.smithi110.stdout:(25/29): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-01-27T03:28:41.086 INFO:teuthology.orchestra.run.smithi110.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 819 kB/s | 192 kB 00:00 2024-01-27T03:28:41.287 INFO:teuthology.orchestra.run.smithi110.stdout:(27/29): ceph-common-19.0.0-926.g5ffe0641.el9.x 6.7 MB/s | 20 MB 00:02 2024-01-27T03:28:41.322 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-01-27T03:28:41.512 INFO:teuthology.orchestra.run.smithi110.stdout:(28/29): librados2-19.0.0-926.g5ffe0641.el9.x86 7.5 MB/s | 3.4 MB 00:00 2024-01-27T03:28:41.570 INFO:teuthology.orchestra.run.smithi110.stdout:(29/29): librbd1-19.0.0-926.g5ffe0641.el9.x86_6 6.5 MB/s | 3.1 MB 00:00 2024-01-27T03:28:41.574 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:28:41.574 INFO:teuthology.orchestra.run.smithi110.stdout:Total 17 MB/s | 57 MB 00:03 2024-01-27T03:28:41.575 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:28:41.607 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:28:41.607 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:28:42.079 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:28:42.079 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:28:43.471 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:28:43.559 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-27T03:28:43.592 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-01-27T03:28:43.641 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-01-27T03:28:44.023 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:28:44.026 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-01-27T03:28:44.026 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 9.7 M 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 3.4 M 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 3.1 M 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 5.1 M 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 20 M 2024-01-27T03:28:44.027 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 25 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 743 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 498 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 5.1 M 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-27T03:28:44.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 45 k 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 130 k 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 161 k 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 320 k 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 298 k 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 99 k 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout:Install 27 Packages 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2024-01-27T03:28:44.029 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:28:44.030 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 57 M 2024-01-27T03:28:44.030 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:28:44.057 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 5/31 2024-01-27T03:28:44.132 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 5/31 2024-01-27T03:28:44.219 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:19.0.0-926.g5ffe0641.el9.x86_64 6/31 2024-01-27T03:28:44.251 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 7/31 2024-01-27T03:28:44.313 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 7/31 2024-01-27T03:28:44.342 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-01-27T03:28:44.385 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:19.0.0-926.g5ffe0641.el9.x 9/31 2024-01-27T03:28:44.453 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:19.0.0-926.g5ffe0641.el9.x86_64 10/31 2024-01-27T03:28:44.478 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:19.0.0-926.g5ffe0641.el9.x86_64 11/31 2024-01-27T03:28:44.543 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:19.0.0-926.g5ffe0641.el9.x86_64 11/31 2024-01-27T03:28:44.650 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-27T03:28:44.685 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-27T03:28:44.725 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-27T03:28:44.766 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-27T03:28:44.795 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-27T03:28:44.828 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-27T03:28:45.145 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-27T03:28:45.174 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 19/31 2024-01-27T03:28:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 19/31 2024-01-27T03:28:45.279 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:19.0.0-926.g5ffe0641.el9.x86_64 20/31 2024-01-27T03:28:45.324 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-27T03:28:45.768 INFO:teuthology.orchestra.run.smithi162.stdout:(1/29): ceph-base-19.0.0-926.g5ffe0641.el9.x86_ 3.7 MB/s | 5.1 MB 00:01 2024-01-27T03:28:45.779 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-27T03:28:45.811 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:19.0.0-926.g5ffe0641.el9.x86_64 23/31 2024-01-27T03:28:45.832 INFO:teuthology.orchestra.run.smithi162.stdout:(2/29): ceph-selinux-19.0.0-926.g5ffe0641.el9.x 379 kB/s | 25 kB 00:00 2024-01-27T03:28:45.865 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:19.0.0-926.g5ffe0641.el9.x86_64 23/31 2024-01-27T03:28:45.942 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:19.0.0-926.g5ffe0641.el9.x86_64 24/31 2024-01-27T03:28:45.972 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:19.0.0-926.g5ffe0641.el9.x86 25/31 2024-01-27T03:28:46.006 INFO:teuthology.orchestra.run.smithi162.stdout:(3/29): libcephfs2-19.0.0-926.g5ffe0641.el9.x86 4.2 MB/s | 743 kB 00:00 2024-01-27T03:28:46.156 INFO:teuthology.orchestra.run.smithi162.stdout:(4/29): libradosstriper1-19.0.0-926.g5ffe0641.e 3.2 MB/s | 498 kB 00:00 2024-01-27T03:28:46.624 INFO:teuthology.orchestra.run.smithi162.stdout:(5/29): ceph-radosgw-19.0.0-926.g5ffe0641.el9.x 4.3 MB/s | 9.7 MB 00:02 2024-01-27T03:28:46.707 INFO:teuthology.orchestra.run.smithi162.stdout:(6/29): python3-ceph-argparse-19.0.0-926.g5ffe0 543 kB/s | 45 kB 00:00 2024-01-27T03:28:46.832 INFO:teuthology.orchestra.run.smithi162.stdout:(7/29): python3-ceph-common-19.0.0-926.g5ffe064 1.0 MB/s | 130 kB 00:00 2024-01-27T03:28:46.941 INFO:teuthology.orchestra.run.smithi162.stdout:(8/29): python3-cephfs-19.0.0-926.g5ffe0641.el9 1.5 MB/s | 161 kB 00:00 2024-01-27T03:28:47.058 INFO:teuthology.orchestra.run.smithi162.stdout:(9/29): python3-rados-19.0.0-926.g5ffe0641.el9. 2.7 MB/s | 320 kB 00:00 2024-01-27T03:28:47.175 INFO:teuthology.orchestra.run.smithi162.stdout:(10/29): python3-rbd-19.0.0-926.g5ffe0641.el9.x 2.5 MB/s | 298 kB 00:00 2024-01-27T03:28:47.275 INFO:teuthology.orchestra.run.smithi162.stdout:(11/29): python3-rgw-19.0.0-926.g5ffe0641.el9.x 997 kB/s | 99 kB 00:00 2024-01-27T03:28:47.383 INFO:teuthology.orchestra.run.smithi162.stdout:(12/29): librgw2-19.0.0-926.g5ffe0641.el9.x86_6 4.2 MB/s | 5.1 MB 00:01 2024-01-27T03:28:47.575 INFO:teuthology.orchestra.run.smithi162.stdout:(13/29): ceph-common-19.0.0-926.g5ffe0641.el9.x 6.2 MB/s | 20 MB 00:03 2024-01-27T03:28:47.634 INFO:teuthology.orchestra.run.smithi162.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 638 kB/s | 159 kB 00:00 2024-01-27T03:28:47.676 INFO:teuthology.orchestra.run.smithi162.stdout:(15/29): boost-program-options-1.75.0-8.el9.x86 264 kB/s | 106 kB 00:00 2024-01-27T03:28:47.701 INFO:teuthology.orchestra.run.smithi162.stdout:(16/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 353 kB/s | 44 kB 00:00 2024-01-27T03:28:47.734 INFO:teuthology.orchestra.run.smithi162.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-01-27T03:28:47.760 INFO:teuthology.orchestra.run.smithi162.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 714 kB/s | 41 kB 00:00 2024-01-27T03:28:47.793 INFO:teuthology.orchestra.run.smithi162.stdout:(19/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-01-27T03:28:47.994 INFO:teuthology.orchestra.run.smithi162.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 172 kB/s | 26 kB 00:00 2024-01-27T03:28:48.021 INFO:teuthology.orchestra.run.smithi162.stdout:(21/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-01-27T03:28:48.044 INFO:teuthology.orchestra.run.smithi162.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 972 kB/s | 49 kB 00:00 2024-01-27T03:28:48.061 INFO:teuthology.orchestra.run.smithi162.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-01-27T03:28:48.137 INFO:teuthology.orchestra.run.smithi162.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-01-27T03:28:48.162 INFO:teuthology.orchestra.run.smithi162.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-01-27T03:28:48.202 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 26/31 2024-01-27T03:28:48.229 INFO:teuthology.orchestra.run.smithi162.stdout:(26/29): libarrow-9.0.0-10.el9.x86_64.rpm 9.4 MB/s | 4.4 MB 00:00 2024-01-27T03:28:48.234 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 26/31 2024-01-27T03:28:48.271 INFO:teuthology.orchestra.run.smithi162.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-01-27T03:28:48.769 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 26/31 2024-01-27T03:28:48.770 INFO:teuthology.orchestra.run.smithi162.stdout:(28/29): librbd1-19.0.0-926.g5ffe0641.el9.x86_6 5.8 MB/s | 3.1 MB 00:00 2024-01-27T03:28:48.812 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 27/31 2024-01-27T03:28:48.898 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 27/31 2024-01-27T03:28:48.898 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-27T03:28:48.898 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-27T03:28:48.898 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:28:48.930 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:19.0.0-926.g5ffe0641.el9.x86_64 28/31 2024-01-27T03:28:49.079 INFO:teuthology.orchestra.run.smithi162.stdout:(29/29): librados2-19.0.0-926.g5ffe0641.el9.x86 3.7 MB/s | 3.4 MB 00:00 2024-01-27T03:28:49.085 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:28:49.085 INFO:teuthology.orchestra.run.smithi162.stdout:Total 11 MB/s | 57 MB 00:05 2024-01-27T03:28:49.086 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:28:49.122 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:28:49.122 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:28:49.597 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:28:49.597 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:28:50.965 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:28:51.050 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-27T03:28:51.084 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-01-27T03:28:51.134 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-01-27T03:28:51.526 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-01-27T03:28:51.557 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 5/31 2024-01-27T03:28:51.648 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 5/31 2024-01-27T03:28:51.736 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.0.0-926.g5ffe0641.el9.x86_64 6/31 2024-01-27T03:28:51.767 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 7/31 2024-01-27T03:28:51.838 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 7/31 2024-01-27T03:28:51.915 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-01-27T03:28:51.986 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.0.0-926.g5ffe0641.el9.x 9/31 2024-01-27T03:28:52.054 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.0.0-926.g5ffe0641.el9.x86_64 10/31 2024-01-27T03:28:52.078 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.0.0-926.g5ffe0641.el9.x86_64 11/31 2024-01-27T03:28:52.144 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.0.0-926.g5ffe0641.el9.x86_64 11/31 2024-01-27T03:28:52.250 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-27T03:28:52.285 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-27T03:28:52.317 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-27T03:28:52.358 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-27T03:28:52.387 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-27T03:28:52.420 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-27T03:28:52.731 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-27T03:28:52.758 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 19/31 2024-01-27T03:28:52.847 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 19/31 2024-01-27T03:28:52.879 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.0.0-926.g5ffe0641.el9.x86_64 20/31 2024-01-27T03:28:52.922 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-27T03:28:53.379 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-27T03:28:53.410 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.0.0-926.g5ffe0641.el9.x86_64 23/31 2024-01-27T03:28:53.481 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.0.0-926.g5ffe0641.el9.x86_64 23/31 2024-01-27T03:28:53.549 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.0.0-926.g5ffe0641.el9.x86_64 24/31 2024-01-27T03:28:53.588 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.0.0-926.g5ffe0641.el9.x86 25/31 2024-01-27T03:28:56.005 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 26/31 2024-01-27T03:28:56.042 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 26/31 2024-01-27T03:28:56.568 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 26/31 2024-01-27T03:28:56.612 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 27/31 2024-01-27T03:28:56.690 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 27/31 2024-01-27T03:28:56.690 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-27T03:28:56.690 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-27T03:28:56.690 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:28:56.738 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.0.0-926.g5ffe0641.el9.x86_64 28/31 2024-01-27T03:28:58.557 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:19.0.0-926.g5ffe0641.el9.x86_64 28/31 2024-01-27T03:28:58.557 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-01-27T03:28:58.557 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-01-27T03:28:58.557 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-01-27T03:28:58.557 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-01-27T03:28:58.557 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-01-27T03:28:58.557 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-01-27T03:28:58.557 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-01-27T03:28:58.558 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:28:59.488 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:19.0.0-926.g5ffe0641.el9.x86_64 29/31 2024-01-27T03:28:59.512 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-926.g5ffe0641.el9.x86_64 29/31 2024-01-27T03:28:59.512 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:28:59.512 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-27T03:28:59.512 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-27T03:28:59.512 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-27T03:28:59.512 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:28:59.514 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-27T03:28:59.598 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-27T03:28:59.599 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 1/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 2/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:19.0.0-926.g5ffe0641.el9.x86_64 3/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:19.0.0-926.g5ffe0641.el9.x86_64 4/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 5/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:19.0.0-926.g5ffe0641.el9.x86_64 6/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:19.0.0-926.g5ffe0641.el9.x86_64 7/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:19.0.0-926.g5ffe0641.el9.x 8/31 2024-01-27T03:29:00.753 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:19.0.0-926.g5ffe0641.el9.x86 9/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:19.0.0-926.g5ffe0641.el9.x86_64 10/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:19.0.0-926.g5ffe0641.el9.x86_64 11/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:19.0.0-926.g5ffe0641.el9.x86_64 12/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:19.0.0-926.g5ffe0641.el9.x86_64 13/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-27T03:29:00.754 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-27T03:29:00.755 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-27T03:29:00.755 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-27T03:29:00.755 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-27T03:29:00.755 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-27T03:29:00.755 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 28/31 2024-01-27T03:29:00.755 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-27T03:29:00.755 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 30/31 2024-01-27T03:29:01.855 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-27T03:29:01.855 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:01.855 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-01-27T03:29:01.855 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.855 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.855 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:29:01.855 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-27T03:29:01.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.855 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.856 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: re2-1:20211101-3.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:01.857 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:02.235 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test 2024-01-27T03:29:02.812 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:27 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:02.972 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:02.973 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:02.973 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:29:02.973 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:02.973 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:29:02.973 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 47 M 2024-01-27T03:29:02.973 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 163 k 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-01-27T03:29:02.974 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:02.975 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 48 M 2024-01-27T03:29:02.975 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 204 M 2024-01-27T03:29:02.975 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:29:03.152 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-01-27T03:29:03.194 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-01-27T03:29:03.236 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-01-27T03:29:03.252 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): libcephsqlite-19.0.0-926.g5ffe0641.el9.x 589 kB/s | 163 kB 00:00 2024-01-27T03:29:03.295 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-01-27T03:29:05.611 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): ceph-test-19.0.0-926.g5ffe0641.el9.x86_6 18 MB/s | 47 MB 00:02 2024-01-27T03:29:05.614 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:05.615 INFO:teuthology.orchestra.run.smithi110.stdout:Total 18 MB/s | 48 MB 00:02 2024-01-27T03:29:05.615 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:29:05.635 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:29:05.635 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:29:06.021 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:29:06.021 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:29:06.243 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.0.0-926.g5ffe0641.el9.x86_64 28/31 2024-01-27T03:29:06.243 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-01-27T03:29:06.243 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-01-27T03:29:06.243 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-01-27T03:29:06.243 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-01-27T03:29:06.243 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-01-27T03:29:06.243 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-01-27T03:29:06.243 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-01-27T03:29:06.243 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:06.594 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:29:06.644 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-27T03:29:06.685 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-27T03:29:06.733 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-27T03:29:06.779 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-27T03:29:06.806 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:19.0.0-926.g5ffe0641.el9.x86_64 5/6 2024-01-27T03:29:07.147 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.0.0-926.g5ffe0641.el9.x86_64 29/31 2024-01-27T03:29:07.172 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-926.g5ffe0641.el9.x86_64 29/31 2024-01-27T03:29:07.173 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:07.173 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-27T03:29:07.173 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-27T03:29:07.173 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-27T03:29:07.173 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:07.174 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-27T03:29:07.258 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-27T03:29:07.258 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-27T03:29:08.443 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-27T03:29:08.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 1/31 2024-01-27T03:29:08.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 2/31 2024-01-27T03:29:08.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.0.0-926.g5ffe0641.el9.x86_64 3/31 2024-01-27T03:29:08.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.0.0-926.g5ffe0641.el9.x86_64 4/31 2024-01-27T03:29:08.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 5/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.0.0-926.g5ffe0641.el9.x86_64 6/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.0.0-926.g5ffe0641.el9.x86_64 7/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.0.0-926.g5ffe0641.el9.x 8/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.0.0-926.g5ffe0641.el9.x86 9/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.0.0-926.g5ffe0641.el9.x86_64 10/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.0.0-926.g5ffe0641.el9.x86_64 11/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.0.0-926.g5ffe0641.el9.x86_64 12/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.0.0-926.g5ffe0641.el9.x86_64 13/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-27T03:29:08.444 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 28/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-27T03:29:08.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 30/31 2024-01-27T03:29:09.506 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-27T03:29:09.506 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:09.506 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-01-27T03:29:09.506 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.506 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-27T03:29:09.507 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2024-01-27T03:29:09.508 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-27T03:29:09.509 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:09.509 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:29:09.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2024-01-27T03:29:10.418 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:29 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:29:10.573 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:29:10.574 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:29:10.574 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:10.574 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:29:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 47 M 2024-01-27T03:29:10.574 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:29:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-27T03:29:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 163 k 2024-01-27T03:29:10.574 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-27T03:29:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-27T03:29:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-27T03:29:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:10.575 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:29:10.575 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:10.575 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-01-27T03:29:10.575 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:10.576 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 48 M 2024-01-27T03:29:10.576 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 204 M 2024-01-27T03:29:10.576 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:29:10.754 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-01-27T03:29:10.787 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-01-27T03:29:10.829 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-01-27T03:29:10.863 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): libcephsqlite-19.0.0-926.g5ffe0641.el9.x 570 kB/s | 163 kB 00:00 2024-01-27T03:29:10.888 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-01-27T03:29:11.303 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:19.0.0-926.g5ffe0641.el9.x86_64 5/6 2024-01-27T03:29:11.380 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:19.0.0-926.g5ffe0641.el9.x86_64 6/6 2024-01-27T03:29:13.113 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): ceph-test-19.0.0-926.g5ffe0641.el9.x86_6 19 MB/s | 47 MB 00:02 2024-01-27T03:29:13.116 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:13.116 INFO:teuthology.orchestra.run.smithi162.stdout:Total 19 MB/s | 48 MB 00:02 2024-01-27T03:29:13.116 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:29:13.136 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:29:13.137 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:29:13.544 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:29:13.544 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:29:13.817 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-2:19.0.0-926.g5ffe0641.el9.x86_64 6/6 2024-01-27T03:29:13.817 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:19.0.0-926.g5ffe0641.el9.x86_64 1/6 2024-01-27T03:29:13.817 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:19.0.0-926.g5ffe0641.el9.x86_64 2/6 2024-01-27T03:29:13.817 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-27T03:29:13.817 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-27T03:29:13.817 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-27T03:29:14.140 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:29:14.194 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-27T03:29:14.210 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-27T03:29:14.210 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:14.210 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:29:14.210 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:14.210 INFO:teuthology.orchestra.run.smithi110.stdout: jq-1.6-16.el9.x86_64 2024-01-27T03:29:14.210 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:14.211 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-27T03:29:14.211 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-27T03:29:14.211 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-27T03:29:14.211 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:14.211 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-27T03:29:14.283 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-27T03:29:14.337 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-27T03:29:14.366 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.0.0-926.g5ffe0641.el9.x86_64 5/6 2024-01-27T03:29:14.484 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph 2024-01-27T03:29:15.064 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:40 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:15.225 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:15.227 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-01-27T03:29:15.227 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:29:15.227 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-01-27T03:29:15.227 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:29:15.227 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 6.5 k 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 2.1 M 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 1.5 M 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 246 k 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 4.7 M 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 17 M 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-27T03:29:15.228 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-27T03:29:15.229 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-27T03:29:15.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-27T03:29:15.231 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:15.231 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:29:15.231 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2024-01-27T03:29:15.231 INFO:teuthology.orchestra.run.smithi110.stdout:Install 36 Packages 2024-01-27T03:29:15.231 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:15.231 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 30 M 2024-01-27T03:29:15.231 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 106 M 2024-01-27T03:29:15.232 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:29:15.764 INFO:teuthology.orchestra.run.smithi110.stdout:(1/36): ceph-19.0.0-926.g5ffe0641.el9.x86_64.rp 28 kB/s | 6.5 kB 00:00 2024-01-27T03:29:16.164 INFO:teuthology.orchestra.run.smithi110.stdout:(2/36): ceph-mds-19.0.0-926.g5ffe0641.el9.x86_6 3.3 MB/s | 2.1 MB 00:00 2024-01-27T03:29:16.206 INFO:teuthology.orchestra.run.smithi110.stdout:(3/36): ceph-mgr-19.0.0-926.g5ffe0641.el9.x86_6 2.2 MB/s | 1.5 MB 00:00 2024-01-27T03:29:16.273 INFO:teuthology.orchestra.run.smithi110.stdout:(4/36): ceph-mgr-modules-core-19.0.0-926.g5ffe0 3.6 MB/s | 246 kB 00:00 2024-01-27T03:29:16.390 INFO:teuthology.orchestra.run.smithi110.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-01-27T03:29:16.432 INFO:teuthology.orchestra.run.smithi110.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-01-27T03:29:16.466 INFO:teuthology.orchestra.run.smithi110.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-01-27T03:29:16.499 INFO:teuthology.orchestra.run.smithi110.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-01-27T03:29:16.533 INFO:teuthology.orchestra.run.smithi110.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-01-27T03:29:16.575 INFO:teuthology.orchestra.run.smithi110.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-01-27T03:29:16.642 INFO:teuthology.orchestra.run.smithi110.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-01-27T03:29:16.684 INFO:teuthology.orchestra.run.smithi110.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-01-27T03:29:16.728 INFO:teuthology.orchestra.run.smithi110.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-01-27T03:29:16.785 INFO:teuthology.orchestra.run.smithi110.stdout:(14/36): ceph-mon-19.0.0-926.g5ffe0641.el9.x86_ 4.6 MB/s | 4.7 MB 00:01 2024-01-27T03:29:16.812 INFO:teuthology.orchestra.run.smithi110.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 1.3 MB/s | 113 kB 00:00 2024-01-27T03:29:16.838 INFO:teuthology.orchestra.run.smithi110.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.5 MB/s | 187 kB 00:00 2024-01-27T03:29:17.238 INFO:teuthology.orchestra.run.smithi110.stdout:(17/36): ceph-osd-19.0.0-926.g5ffe0641.el9.x86_ 16 MB/s | 17 MB 00:01 2024-01-27T03:29:17.265 INFO:teuthology.orchestra.run.smithi110.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 49 kB/s | 22 kB 00:00 2024-01-27T03:29:17.606 INFO:teuthology.orchestra.run.smithi110.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 197 kB/s | 151 kB 00:00 2024-01-27T03:29:17.772 INFO:teuthology.orchestra.run.smithi110.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 81 kB/s | 43 kB 00:00 2024-01-27T03:29:17.806 INFO:teuthology.orchestra.run.smithi110.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-01-27T03:29:17.881 INFO:teuthology.orchestra.run.smithi110.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 278 kB/s | 172 kB 00:00 2024-01-27T03:29:17.906 INFO:teuthology.orchestra.run.smithi110.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-01-27T03:29:17.932 INFO:teuthology.orchestra.run.smithi110.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2024-01-27T03:29:17.999 INFO:teuthology.orchestra.run.smithi110.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 197 kB/s | 23 kB 00:00 2024-01-27T03:29:18.024 INFO:teuthology.orchestra.run.smithi110.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-01-27T03:29:18.050 INFO:teuthology.orchestra.run.smithi110.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2024-01-27T03:29:18.099 INFO:teuthology.orchestra.run.smithi110.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2024-01-27T03:29:18.158 INFO:teuthology.orchestra.run.smithi110.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-01-27T03:29:18.200 INFO:teuthology.orchestra.run.smithi110.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-01-27T03:29:18.225 INFO:teuthology.orchestra.run.smithi110.stdout:(31/36): python3-more-itertools-8.12.0-2.el9.no 392 kB/s | 79 kB 00:00 2024-01-27T03:29:18.275 INFO:teuthology.orchestra.run.smithi110.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 766 kB/s | 90 kB 00:00 2024-01-27T03:29:18.309 INFO:teuthology.orchestra.run.smithi110.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 328 kB/s | 36 kB 00:00 2024-01-27T03:29:18.392 INFO:teuthology.orchestra.run.smithi110.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-01-27T03:29:18.418 INFO:teuthology.orchestra.run.smithi110.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-01-27T03:29:18.443 INFO:teuthology.orchestra.run.smithi110.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 149 kB/s | 20 kB 00:00 2024-01-27T03:29:18.449 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:18.449 INFO:teuthology.orchestra.run.smithi110.stdout:Total 9.2 MB/s | 30 MB 00:03 2024-01-27T03:29:18.463 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:29:18.504 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:29:18.504 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:29:18.895 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.0.0-926.g5ffe0641.el9.x86_64 5/6 2024-01-27T03:29:18.919 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:29:18.920 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:29:18.973 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.0.0-926.g5ffe0641.el9.x86_64 6/6 2024-01-27T03:29:20.259 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:29:20.321 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-27T03:29:20.354 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-27T03:29:20.382 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-27T03:29:20.452 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-27T03:29:20.497 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-27T03:29:20.530 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-27T03:29:20.622 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-27T03:29:20.708 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-27T03:29:20.752 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-27T03:29:20.820 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-27T03:29:20.875 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-27T03:29:20.913 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-27T03:29:20.965 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-27T03:29:21.020 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-27T03:29:21.204 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-27T03:29:21.254 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-27T03:29:21.284 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-27T03:29:21.319 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-27T03:29:21.349 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-27T03:29:21.425 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-27T03:29:21.460 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-27T03:29:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:19.0.0-926.g5ffe0641.el9.x86_64 6/6 2024-01-27T03:29:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.0.0-926.g5ffe0641.el9.x86_64 1/6 2024-01-27T03:29:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.0.0-926.g5ffe0641.el9.x86_64 2/6 2024-01-27T03:29:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-27T03:29:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-27T03:29:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-27T03:29:21.493 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-27T03:29:21.533 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-27T03:29:21.568 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-27T03:29:21.603 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-27T03:29:21.692 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-27T03:29:21.726 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-27T03:29:21.806 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-27T03:29:21.877 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-16.el9.x86_64 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:21.878 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:29:21.943 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-27T03:29:22.060 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-27T03:29:22.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2024-01-27T03:29:22.226 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-926.g5ffe0641.el9.n 31/36 2024-01-27T03:29:22.257 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 32/36 2024-01-27T03:29:22.281 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 32/36 2024-01-27T03:29:22.282 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:22.282 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-27T03:29:22.282 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-27T03:29:22.282 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-27T03:29:22.282 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:22.761 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:41 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:29:22.920 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:29:22.922 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-01-27T03:29:22.922 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:29:22.922 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-01-27T03:29:22.922 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:29:22.922 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 6.5 k 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 2.1 M 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 1.5 M 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 246 k 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 4.7 M 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 17 M 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-27T03:29:22.923 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-27T03:29:22.924 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout:Install 36 Packages 2024-01-27T03:29:22.925 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:22.926 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 30 M 2024-01-27T03:29:22.926 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 106 M 2024-01-27T03:29:22.926 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:29:23.290 INFO:teuthology.orchestra.run.smithi162.stdout:(1/36): ceph-19.0.0-926.g5ffe0641.el9.x86_64.rp 40 kB/s | 6.5 kB 00:00 2024-01-27T03:29:23.690 INFO:teuthology.orchestra.run.smithi162.stdout:(2/36): ceph-mon-19.0.0-926.g5ffe0641.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-01-27T03:29:23.774 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:19.0.0-926.g5ffe0641.el9.x86_64 33/36 2024-01-27T03:29:23.802 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:19.0.0-926.g5ffe0641.el9.x86_64 33/36 2024-01-27T03:29:23.802 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:23.802 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-27T03:29:23.802 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-27T03:29:23.803 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-27T03:29:23.803 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:23.966 INFO:teuthology.orchestra.run.smithi162.stdout:(3/36): ceph-mgr-19.0.0-926.g5ffe0641.el9.x86_6 1.8 MB/s | 1.5 MB 00:00 2024-01-27T03:29:24.099 INFO:teuthology.orchestra.run.smithi162.stdout:(4/36): ceph-mgr-modules-core-19.0.0-926.g5ffe0 1.8 MB/s | 246 kB 00:00 2024-01-27T03:29:24.215 INFO:teuthology.orchestra.run.smithi162.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-01-27T03:29:24.242 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:19.0.0-926.g5ffe0641.el9.x86_64 34/36 2024-01-27T03:29:24.256 INFO:teuthology.orchestra.run.smithi162.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-01-27T03:29:24.265 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:19.0.0-926.g5ffe0641.el9.x86_64 34/36 2024-01-27T03:29:24.265 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:24.265 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-27T03:29:24.265 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-27T03:29:24.265 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-27T03:29:24.266 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:24.298 INFO:teuthology.orchestra.run.smithi162.stdout:(7/36): ceph-mds-19.0.0-926.g5ffe0641.el9.x86_6 1.8 MB/s | 2.1 MB 00:01 2024-01-27T03:29:24.324 INFO:teuthology.orchestra.run.smithi162.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 470 kB/s | 32 kB 00:00 2024-01-27T03:29:24.349 INFO:teuthology.orchestra.run.smithi162.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 933 kB/s | 47 kB 00:00 2024-01-27T03:29:24.375 INFO:teuthology.orchestra.run.smithi162.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2024-01-27T03:29:24.400 INFO:teuthology.orchestra.run.smithi162.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-01-27T03:29:24.450 INFO:teuthology.orchestra.run.smithi162.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-01-27T03:29:24.468 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:19.0.0-926.g5ffe0641.el9.x86_64 35/36 2024-01-27T03:29:24.475 INFO:teuthology.orchestra.run.smithi162.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-01-27T03:29:24.490 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:19.0.0-926.g5ffe0641.el9.x86_64 35/36 2024-01-27T03:29:24.490 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:24.490 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-27T03:29:24.490 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-27T03:29:24.491 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-27T03:29:24.491 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:24.501 INFO:teuthology.orchestra.run.smithi162.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-27T03:29:24.526 INFO:teuthology.orchestra.run.smithi162.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-27T03:29:24.557 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:19.0.0-926.g5ffe0641.el9.x86_64 36/36 2024-01-27T03:29:24.559 INFO:teuthology.orchestra.run.smithi162.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-01-27T03:29:24.735 INFO:teuthology.orchestra.run.smithi162.stdout:(17/36): ceph-osd-19.0.0-926.g5ffe0641.el9.x86_ 16 MB/s | 17 MB 00:01 2024-01-27T03:29:24.911 INFO:teuthology.orchestra.run.smithi162.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 288 kB/s | 43 kB 00:00 2024-01-27T03:29:24.936 INFO:teuthology.orchestra.run.smithi162.stdout:(19/36): lua-devel-5.4.4-4.el9.x86_64.rpm 54 kB/s | 22 kB 00:00 2024-01-27T03:29:24.961 INFO:teuthology.orchestra.run.smithi162.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 377 kB/s | 151 kB 00:00 2024-01-27T03:29:25.028 INFO:teuthology.orchestra.run.smithi162.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-01-27T03:29:25.053 INFO:teuthology.orchestra.run.smithi162.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-01-27T03:29:25.129 INFO:teuthology.orchestra.run.smithi162.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 179 kB/s | 18 kB 00:00 2024-01-27T03:29:25.154 INFO:teuthology.orchestra.run.smithi162.stdout:(24/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 55 kB/s | 11 kB 00:00 2024-01-27T03:29:25.179 INFO:teuthology.orchestra.run.smithi162.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 185 kB/s | 23 kB 00:00 2024-01-27T03:29:25.204 INFO:teuthology.orchestra.run.smithi162.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-01-27T03:29:25.230 INFO:teuthology.orchestra.run.smithi162.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-01-27T03:29:25.256 INFO:teuthology.orchestra.run.smithi162.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-01-27T03:29:25.280 INFO:teuthology.orchestra.run.smithi162.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-01-27T03:29:25.305 INFO:teuthology.orchestra.run.smithi162.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-01-27T03:29:25.331 INFO:teuthology.orchestra.run.smithi162.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-01-27T03:29:25.356 INFO:teuthology.orchestra.run.smithi162.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-01-27T03:29:25.381 INFO:teuthology.orchestra.run.smithi162.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-01-27T03:29:25.440 INFO:teuthology.orchestra.run.smithi162.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-01-27T03:29:25.465 INFO:teuthology.orchestra.run.smithi162.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-01-27T03:29:25.515 INFO:teuthology.orchestra.run.smithi162.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-01-27T03:29:25.520 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:25.521 INFO:teuthology.orchestra.run.smithi162.stdout:Total 11 MB/s | 30 MB 00:02 2024-01-27T03:29:25.534 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:29:25.574 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:29:25.574 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:29:25.976 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:29:25.977 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:29:26.288 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-2:19.0.0-926.g5ffe0641.el9.x86_64 36/36 2024-01-27T03:29:26.288 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:19.0.0-926.g5ffe0641.el9.x86_64 1/36 2024-01-27T03:29:26.288 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:19.0.0-926.g5ffe0641.el9.x86_64 2/36 2024-01-27T03:29:26.288 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 3/36 2024-01-27T03:29:26.288 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:19.0.0-926.g5ffe0641.el9.x86_64 4/36 2024-01-27T03:29:26.289 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:19.0.0-926.g5ffe0641.el9.x86_64 5/36 2024-01-27T03:29:26.289 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-926.g5ffe0641.el9.n 6/36 2024-01-27T03:29:26.289 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-27T03:29:26.289 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-27T03:29:26.289 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-27T03:29:26.289 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-27T03:29:26.289 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-27T03:29:26.290 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-27T03:29:26.290 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-27T03:29:26.290 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-27T03:29:26.290 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-27T03:29:26.290 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-27T03:29:26.290 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-27T03:29:26.291 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-27T03:29:26.291 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-27T03:29:26.291 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-27T03:29:26.291 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-27T03:29:26.291 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-27T03:29:26.291 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-27T03:29:26.291 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-27T03:29:26.291 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-27T03:29:26.292 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-27T03:29:26.292 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-27T03:29:26.292 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-27T03:29:26.292 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-27T03:29:26.292 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-27T03:29:26.292 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-27T03:29:26.292 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-27T03:29:26.292 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-27T03:29:26.293 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-27T03:29:26.293 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: lua-5.4.4-4.el9.x86_64 2024-01-27T03:29:27.155 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-27T03:29:27.156 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-27T03:29:27.157 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-27T03:29:27.158 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:27.158 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:27.295 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:29:27.351 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-27T03:29:27.393 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-27T03:29:27.420 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-27T03:29:27.491 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-27T03:29:27.514 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base 2024-01-27T03:29:27.528 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-27T03:29:27.561 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-27T03:29:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-27T03:29:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-27T03:29:27.781 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-27T03:29:27.854 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-27T03:29:27.908 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-27T03:29:27.946 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-27T03:29:28.004 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-27T03:29:28.058 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-27T03:29:28.089 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:53 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:28.189 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:29:28.238 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:28.239 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:29:28.239 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-27T03:29:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-27T03:29:28.304 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install cephadm 2024-01-27T03:29:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-27T03:29:28.374 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-27T03:29:28.404 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-27T03:29:28.481 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-27T03:29:28.514 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-27T03:29:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-27T03:29:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-27T03:29:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-27T03:29:28.683 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-27T03:29:28.773 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-27T03:29:28.822 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-27T03:29:28.874 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:53 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:28.894 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-27T03:29:29.026 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:29.026 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:29.027 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:29:29.027 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:29.027 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:29:29.027 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 742 k 2024-01-27T03:29:29.027 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:29.027 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:29:29.027 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:29.027 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:29:29.027 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:29.028 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 742 k 2024-01-27T03:29:29.028 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 748 k 2024-01-27T03:29:29.028 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:29:29.041 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-27T03:29:29.175 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-27T03:29:29.349 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-926.g5ffe0641.el9.n 31/36 2024-01-27T03:29:29.377 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 32/36 2024-01-27T03:29:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 32/36 2024-01-27T03:29:29.403 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:29.403 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-27T03:29:29.403 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-27T03:29:29.403 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-27T03:29:29.403 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:29.617 INFO:teuthology.orchestra.run.smithi110.stdout:cephadm-19.0.0-926.g5ffe0641.el9.noarch.rpm 1.2 MB/s | 742 kB 00:00 2024-01-27T03:29:29.618 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:29.618 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.2 MB/s | 742 kB 00:00 2024-01-27T03:29:29.618 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:29:29.623 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:29:29.624 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:29:29.629 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:29:29.629 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:29:29.706 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:29:30.163 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:29:30.238 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:29:30.625 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:29:30.836 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.0.0-926.g5ffe0641.el9.x86_64 33/36 2024-01-27T03:29:30.862 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.0.0-926.g5ffe0641.el9.x86_64 33/36 2024-01-27T03:29:30.862 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:30.862 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-27T03:29:30.862 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-27T03:29:30.862 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-27T03:29:30.863 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:30.910 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:29:30.910 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:30.910 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:29:30.910 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:30.910 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:30.910 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:31.120 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache 2024-01-27T03:29:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.0.0-926.g5ffe0641.el9.x86_64 34/36 2024-01-27T03:29:31.327 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.0.0-926.g5ffe0641.el9.x86_64 34/36 2024-01-27T03:29:31.327 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:31.327 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-27T03:29:31.327 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-27T03:29:31.327 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-27T03:29:31.327 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:31.529 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.0.0-926.g5ffe0641.el9.x86_64 35/36 2024-01-27T03:29:31.552 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.0.0-926.g5ffe0641.el9.x86_64 35/36 2024-01-27T03:29:31.552 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:31.553 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-27T03:29:31.553 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-27T03:29:31.553 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-27T03:29:31.553 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:31.618 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.0.0-926.g5ffe0641.el9.x86_64 36/36 2024-01-27T03:29:31.704 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:56 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:31.859 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:31.860 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:31.860 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:29:31.860 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:31.860 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:29:31.860 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 145 k 2024-01-27T03:29:31.860 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:31.861 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:29:31.861 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:31.861 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:29:31.861 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:31.861 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 145 k 2024-01-27T03:29:31.861 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 436 k 2024-01-27T03:29:31.861 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:29:32.231 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-19.0.0-926.g5ffe064 392 kB/s | 145 kB 00:00 2024-01-27T03:29:32.232 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:32.232 INFO:teuthology.orchestra.run.smithi110.stdout:Total 390 kB/s | 145 kB 00:00 2024-01-27T03:29:32.232 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:29:32.240 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:29:32.240 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:29:32.291 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:29:32.291 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:29:32.425 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:29:32.482 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-926.g5ffe0641.e 1/1 2024-01-27T03:29:32.506 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-926.g5ffe0641.e 1/1 2024-01-27T03:29:32.506 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:32.506 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-27T03:29:32.506 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:33.467 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-926.g5ffe0641.e 1/1 2024-01-27T03:29:33.467 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:33.467 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:29:33.467 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:33.467 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:33.467 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:19.0.0-926.g5ffe0641.el9.x86_64 36/36 2024-01-27T03:29:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.0.0-926.g5ffe0641.el9.x86_64 1/36 2024-01-27T03:29:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.0.0-926.g5ffe0641.el9.x86_64 2/36 2024-01-27T03:29:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 3/36 2024-01-27T03:29:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.0.0-926.g5ffe0641.el9.x86_64 4/36 2024-01-27T03:29:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.0.0-926.g5ffe0641.el9.x86_64 5/36 2024-01-27T03:29:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-926.g5ffe0641.el9.n 6/36 2024-01-27T03:29:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-27T03:29:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-27T03:29:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-27T03:29:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-27T03:29:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-27T03:29:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-27T03:29:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-27T03:29:33.727 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr 2024-01-27T03:29:34.319 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:59 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:34.420 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:29:34.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-27T03:29:34.440 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:34.440 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:29:34.440 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:34.440 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:34.440 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-27T03:29:34.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-27T03:29:34.442 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-27T03:29:34.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-27T03:29:34.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-27T03:29:34.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-27T03:29:34.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-27T03:29:34.443 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-27T03:29:34.443 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:34.443 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:29:34.470 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:34.471 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:29:34.471 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:34.547 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-dashboard 2024-01-27T03:29:34.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2024-01-27T03:29:35.128 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:00 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:35.280 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:35.281 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:35.281 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:29:35.281 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:35.281 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:29:35.281 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 1.7 M 2024-01-27T03:29:35.281 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:29:35.281 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 28 k 2024-01-27T03:29:35.281 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 15 k 2024-01-27T03:29:35.282 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-27T03:29:35.282 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-27T03:29:35.282 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:35.282 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:29:35.282 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:35.282 INFO:teuthology.orchestra.run.smithi110.stdout:Install 5 Packages 2024-01-27T03:29:35.282 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:35.283 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.0 M 2024-01-27T03:29:35.283 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 8.5 M 2024-01-27T03:29:35.283 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:29:35.384 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:54 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:29:35.488 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:29:35.528 INFO:teuthology.orchestra.run.smithi110.stdout:(1/5): ceph-prometheus-alerts-19.0.0-926.g5ffe0 78 kB/s | 15 kB 00:00 2024-01-27T03:29:35.537 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:29:35.538 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:29:35.538 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:29:35.553 INFO:teuthology.orchestra.run.smithi110.stdout:(2/5): ceph-grafana-dashboards-19.0.0-926.g5ffe 129 kB/s | 28 kB 00:00 2024-01-27T03:29:35.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2024-01-27T03:29:35.895 INFO:teuthology.orchestra.run.smithi110.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 84 kB/s | 31 kB 00:00 2024-01-27T03:29:36.062 INFO:teuthology.orchestra.run.smithi110.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 371 kB/s | 188 kB 00:00 2024-01-27T03:29:36.201 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:55 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:29:36.353 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:29:36.354 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:36.354 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:29:36.354 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 742 k 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 742 k 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 748 k 2024-01-27T03:29:36.355 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:29:36.670 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-19.0.0-926.g5ffe0641.el9.noarch.rpm 2.3 MB/s | 742 kB 00:00 2024-01-27T03:29:36.670 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:36.670 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.3 MB/s | 742 kB 00:00 2024-01-27T03:29:36.671 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:29:36.676 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:29:36.676 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:29:36.679 INFO:teuthology.orchestra.run.smithi110.stdout:(5/5): ceph-mgr-dashboard-19.0.0-926.g5ffe0641. 1.3 MB/s | 1.7 MB 00:01 2024-01-27T03:29:36.681 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:36.681 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-01-27T03:29:36.681 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:29:36.682 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:29:36.682 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:29:36.689 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:29:36.689 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:29:36.756 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:29:36.756 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:29:36.758 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:29:37.022 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:29:37.118 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-27T03:29:37.159 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-27T03:29:37.173 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:29:37.188 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-926.g5ffe0641.el9.no 3/5 2024-01-27T03:29:37.250 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:29:37.422 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-926.g5ffe0641.el9.n 4/5 2024-01-27T03:29:37.465 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-926.g5ffe0641.el9.noarch 5/5 2024-01-27T03:29:37.694 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:29:37.994 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-926.g5ffe0641.el9.noarch 5/5 2024-01-27T03:29:37.995 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-926.g5ffe0641.el9.n 1/5 2024-01-27T03:29:37.995 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-926.g5ffe0641.el9.noarch 2/5 2024-01-27T03:29:37.995 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-926.g5ffe0641.el9.no 3/5 2024-01-27T03:29:37.995 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-27T03:29:38.043 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:29:38.044 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:38.044 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:29:38.044 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:38.044 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:38.044 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:29:38.287 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-27T03:29:38.287 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:38.287 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:29:38.288 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:38.288 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:38.288 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:38.288 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-27T03:29:38.288 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-27T03:29:38.288 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:38.288 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:38.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2024-01-27T03:29:38.571 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-27T03:29:38.894 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:57 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:29:39.048 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:29:39.049 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:39.049 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:29:39.049 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:39.049 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:29:39.050 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 145 k 2024-01-27T03:29:39.050 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:29:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:29:39.050 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 145 k 2024-01-27T03:29:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 436 k 2024-01-27T03:29:39.050 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:29:39.140 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:04 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:39.270 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-19.0.0-926.g5ffe064 661 kB/s | 145 kB 00:00 2024-01-27T03:29:39.270 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:39.271 INFO:teuthology.orchestra.run.smithi162.stdout:Total 656 kB/s | 145 kB 00:00 2024-01-27T03:29:39.271 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:29:39.278 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:29:39.279 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:29:39.292 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:39.293 INFO:teuthology.orchestra.run.smithi110.stdout:=========================================================================================== 2024-01-27T03:29:39.293 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:29:39.293 INFO:teuthology.orchestra.run.smithi110.stdout:=========================================================================================== 2024-01-27T03:29:39.293 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 7.4 M 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-27T03:29:39.294 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:39.295 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:29:39.295 INFO:teuthology.orchestra.run.smithi110.stdout:=========================================================================================== 2024-01-27T03:29:39.295 INFO:teuthology.orchestra.run.smithi110.stdout:Install 12 Packages 2024-01-27T03:29:39.295 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:39.295 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 37 M 2024-01-27T03:29:39.295 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 209 M 2024-01-27T03:29:39.296 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:29:39.330 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:29:39.330 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:29:39.472 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:29:39.529 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-926.g5ffe0641.e 1/1 2024-01-27T03:29:39.554 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-926.g5ffe0641.e 1/1 2024-01-27T03:29:39.554 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:29:39.554 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-27T03:29:39.554 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:39.719 INFO:teuthology.orchestra.run.smithi110.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 229 kB/s | 32 kB 00:00 2024-01-27T03:29:39.752 INFO:teuthology.orchestra.run.smithi110.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 541 kB/s | 17 kB 00:00 2024-01-27T03:29:39.786 INFO:teuthology.orchestra.run.smithi110.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-01-27T03:29:39.903 INFO:teuthology.orchestra.run.smithi110.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.3 MB/s | 3.0 MB 00:00 2024-01-27T03:29:40.003 INFO:teuthology.orchestra.run.smithi110.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-01-27T03:29:40.095 INFO:teuthology.orchestra.run.smithi110.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-01-27T03:29:40.120 INFO:teuthology.orchestra.run.smithi110.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2024-01-27T03:29:40.262 INFO:teuthology.orchestra.run.smithi110.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 802 kB 00:00 2024-01-27T03:29:40.496 INFO:teuthology.orchestra.run.smithi110.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-01-27T03:29:40.517 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-926.g5ffe0641.e 1/1 2024-01-27T03:29:40.517 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:40.517 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:29:40.517 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:29:40.518 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:40.518 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:29:40.521 INFO:teuthology.orchestra.run.smithi110.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 757 kB/s | 194 kB 00:00 2024-01-27T03:29:40.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2024-01-27T03:29:41.130 INFO:teuthology.orchestra.run.smithi110.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 388 kB/s | 245 kB 00:00 2024-01-27T03:29:41.363 INFO:teuthology.orchestra.run.smithi110.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-9 4.1 MB/s | 7.4 MB 00:01 2024-01-27T03:29:41.365 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:00 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:29:41.367 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:41.367 INFO:teuthology.orchestra.run.smithi110.stdout:Total 18 MB/s | 37 MB 00:02 2024-01-27T03:29:41.379 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:29:41.402 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:29:41.402 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:29:41.469 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:29:41.520 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:29:41.521 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:29:41.522 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:29:41.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-dashboard 2024-01-27T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:29:42.178 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:01 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:29:42.328 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:29:42.329 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:42.329 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:29:42.329 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:42.329 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:29:42.329 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 1.7 M 2024-01-27T03:29:42.329 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:29:42.329 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 28 k 2024-01-27T03:29:42.329 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 15 k 2024-01-27T03:29:42.330 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-27T03:29:42.330 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-27T03:29:42.330 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:42.330 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:29:42.330 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:29:42.330 INFO:teuthology.orchestra.run.smithi162.stdout:Install 5 Packages 2024-01-27T03:29:42.330 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:42.331 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.0 M 2024-01-27T03:29:42.331 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 8.5 M 2024-01-27T03:29:42.331 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:29:42.464 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:29:42.536 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-27T03:29:42.585 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-27T03:29:42.680 INFO:teuthology.orchestra.run.smithi162.stdout:(1/5): ceph-grafana-dashboards-19.0.0-926.g5ffe 145 kB/s | 28 kB 00:00 2024-01-27T03:29:42.681 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-27T03:29:42.700 INFO:teuthology.orchestra.run.smithi162.stdout:(2/5): ceph-prometheus-alerts-19.0.0-926.g5ffe0 68 kB/s | 15 kB 00:00 2024-01-27T03:29:42.713 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-27T03:29:42.882 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-27T03:29:42.883 INFO:teuthology.orchestra.run.smithi162.stdout:(3/5): ceph-mgr-dashboard-19.0.0-926.g5ffe0641. 4.3 MB/s | 1.7 MB 00:00 2024-01-27T03:29:42.920 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-27T03:29:43.001 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-27T03:29:43.158 INFO:teuthology.orchestra.run.smithi162.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 64 kB/s | 31 kB 00:00 2024-01-27T03:29:43.425 INFO:teuthology.orchestra.run.smithi162.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 260 kB/s | 188 kB 00:00 2024-01-27T03:29:43.428 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:43.428 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-01-27T03:29:43.429 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:29:43.436 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:29:43.436 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:29:43.500 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:29:43.500 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:29:43.642 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-27T03:29:43.754 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:29:43.758 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-27T03:29:43.839 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-27T03:29:43.871 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-27T03:29:43.902 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-926.g5ffe0641.el9.no 3/5 2024-01-27T03:29:44.139 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-926.g5ffe0641.el9.n 4/5 2024-01-27T03:29:44.178 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-926.g5ffe0641.el9.noarch 5/5 2024-01-27T03:29:44.701 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-926.g5ffe0641.el9.noarch 5/5 2024-01-27T03:29:44.701 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-926.g5ffe0641.el9.n 1/5 2024-01-27T03:29:44.701 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-926.g5ffe0641.el9.noarch 2/5 2024-01-27T03:29:44.701 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-926.g5ffe0641.el9.no 3/5 2024-01-27T03:29:44.701 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-27T03:29:45.006 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-27T03:29:45.006 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:45.006 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:29:45.006 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:45.006 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:45.006 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:45.007 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-27T03:29:45.007 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-27T03:29:45.007 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:45.007 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:29:45.007 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-27T03:29:45.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-27T03:29:45.801 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:04 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:29:45.956 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout:=========================================================================================== 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout:=========================================================================================== 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 7.4 M 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-27T03:29:45.957 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout:=========================================================================================== 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-01-27T03:29:45.958 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:45.959 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 37 M 2024-01-27T03:29:45.959 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 209 M 2024-01-27T03:29:45.959 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:29:46.032 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-27T03:29:46.067 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-926.g5ffe06 12/12 2024-01-27T03:29:46.390 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 209 kB/s | 32 kB 00:00 2024-01-27T03:29:46.432 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 426 kB/s | 17 kB 00:00 2024-01-27T03:29:46.490 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-01-27T03:29:46.515 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 419 kB/s | 34 kB 00:00 2024-01-27T03:29:46.699 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-01-27T03:29:46.774 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-01-27T03:29:46.808 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-01-27T03:29:46.858 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-01-27T03:29:46.900 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 194 kB 00:00 2024-01-27T03:29:47.192 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-01-27T03:29:47.217 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 779 kB/s | 245 kB 00:00 2024-01-27T03:29:47.784 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-9 4.8 MB/s | 7.4 MB 00:01 2024-01-27T03:29:47.786 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:47.787 INFO:teuthology.orchestra.run.smithi162.stdout:Total 20 MB/s | 37 MB 00:01 2024-01-27T03:29:47.799 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:29:47.823 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:29:47.823 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:29:48.228 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:29:48.229 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:29:48.550 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-926.g5ffe06 12/12 2024-01-27T03:29:48.550 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-926.g5ffe06 1/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-27T03:29:48.551 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-27T03:29:48.931 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:29:49.007 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-27T03:29:49.033 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-27T03:29:49.033 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:49.033 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:29:49.033 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:49.033 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-27T03:29:49.033 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-27T03:29:49.033 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-27T03:29:49.033 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-27T03:29:49.033 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-27T03:29:49.034 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-27T03:29:49.034 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-27T03:29:49.034 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-27T03:29:49.034 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-27T03:29:49.034 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-27T03:29:49.034 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-27T03:29:49.034 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:49.034 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:49.038 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-27T03:29:49.131 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-27T03:29:49.184 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-27T03:29:49.362 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-27T03:29:49.391 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-27T03:29:49.405 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-rook 2024-01-27T03:29:49.473 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-27T03:29:49.994 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:14 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:50.148 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:50.149 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-27T03:29:50.149 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:29:50.149 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-27T03:29:50.149 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 52 k 2024-01-27T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-27T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-27T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.26.2-1.el9 epel 202 k 2024-01-27T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-27T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-27T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-27T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-27T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-27T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-27T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-27T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-27T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-27T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-27T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stdout:Install 13 Packages 2024-01-27T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:50.153 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.0 M 2024-01-27T03:29:50.153 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 26 M 2024-01-27T03:29:50.154 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:29:50.162 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-27T03:29:50.278 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-27T03:29:50.427 INFO:teuthology.orchestra.run.smithi110.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 183 kB/s | 24 kB 00:00 2024-01-27T03:29:50.452 INFO:teuthology.orchestra.run.smithi110.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-01-27T03:29:50.519 INFO:teuthology.orchestra.run.smithi110.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-01-27T03:29:50.536 INFO:teuthology.orchestra.run.smithi110.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-01-27T03:29:50.561 INFO:teuthology.orchestra.run.smithi110.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-01-27T03:29:50.578 INFO:teuthology.orchestra.run.smithi110.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-01-27T03:29:50.596 INFO:teuthology.orchestra.run.smithi110.stdout:(7/13): ceph-mgr-rook-19.0.0-926.g5ffe0641.el9. 172 kB/s | 52 kB 00:00 2024-01-27T03:29:50.746 INFO:teuthology.orchestra.run.smithi110.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-01-27T03:29:50.763 INFO:teuthology.orchestra.run.smithi110.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 81 kB/s | 15 kB 00:00 2024-01-27T03:29:50.821 INFO:teuthology.orchestra.run.smithi110.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-01-27T03:29:50.846 INFO:teuthology.orchestra.run.smithi110.stdout:(11/13): python3-google-auth-2.26.2-1.el9.noarc 986 kB/s | 202 kB 00:00 2024-01-27T03:29:50.880 INFO:teuthology.orchestra.run.smithi110.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-01-27T03:29:51.047 INFO:teuthology.orchestra.run.smithi110.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.4 MB/s | 1.0 MB 00:00 2024-01-27T03:29:51.051 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:51.051 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-01-27T03:29:51.052 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:29:51.064 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:29:51.064 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:29:51.160 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:29:51.160 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:29:51.537 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-27T03:29:51.674 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:29:51.833 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-27T03:29:51.881 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-27T03:29:51.945 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-27T03:29:51.983 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-27T03:29:52.024 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-27T03:29:52.116 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-27T03:29:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.26.2-1.el9.noarch 7/13 2024-01-27T03:29:52.308 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-27T03:29:52.583 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-27T03:29:52.607 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-926.g5ffe06 12/12 2024-01-27T03:29:53.019 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-27T03:29:53.073 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-27T03:29:53.107 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-27T03:29:53.155 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-27T03:29:53.177 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:19.0.0-926.g5ffe0641.el9.noarch 13/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-926.g5ffe0641.el9.noarch 13/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:19.0.0-926.g5ffe0641.el9.noarch 1/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-27T03:29:53.993 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.26.2-1.el9.noarch 10/13 2024-01-27T03:29:53.994 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-27T03:29:53.994 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-27T03:29:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-27T03:29:54.397 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:54.397 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.26.2-1.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-27T03:29:54.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:54.399 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:54.681 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-cephadm 2024-01-27T03:29:55.111 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-926.g5ffe06 12/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-926.g5ffe06 1/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-27T03:29:55.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-27T03:29:55.283 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:20 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:55.438 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 133 k 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:55.439 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:29:55.440 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-27T03:29:55.440 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-01-27T03:29:55.440 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:55.440 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 6.8 M 2024-01-27T03:29:55.440 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 33 M 2024-01-27T03:29:55.441 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:29:55.527 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-27T03:29:55.527 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:55.527 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:29:55.527 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:55.527 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-27T03:29:55.527 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-27T03:29:55.527 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-27T03:29:55.527 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-27T03:29:55.528 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-27T03:29:55.528 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-27T03:29:55.528 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-27T03:29:55.528 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-27T03:29:55.528 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-27T03:29:55.528 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-27T03:29:55.528 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-27T03:29:55.528 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:55.528 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:29:55.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-rook 2024-01-27T03:29:55.959 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-27T03:29:56.109 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-27T03:29:56.135 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): ceph-mgr-cephadm-19.0.0-926.g5ffe0641.el 377 kB/s | 133 kB 00:00 2024-01-27T03:29:56.210 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 575 kB/s | 58 kB 00:00 2024-01-27T03:29:56.302 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-01-27T03:29:56.319 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 278 kB/s | 51 kB 00:00 2024-01-27T03:29:56.320 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:56.321 INFO:teuthology.orchestra.run.smithi110.stdout:Total 7.7 MB/s | 6.8 MB 00:00 2024-01-27T03:29:56.321 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:29:56.330 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:29:56.330 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:29:56.418 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:29:56.418 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:29:56.466 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:15 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:29:56.631 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:29:56.632 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-01-27T03:29:56.632 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:29:56.632 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-01-27T03:29:56.632 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:29:56.632 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 52 k 2024-01-27T03:29:56.632 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:29:56.632 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-27T03:29:56.632 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-27T03:29:56.632 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.26.2-1.el9 epel 202 k 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2024-01-27T03:29:56.633 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:29:56.634 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.0 M 2024-01-27T03:29:56.634 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 26 M 2024-01-27T03:29:56.634 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:29:56.707 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:29:56.864 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-27T03:29:56.911 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-27T03:29:57.108 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 24 kB 00:00 2024-01-27T03:29:57.133 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2024-01-27T03:29:57.192 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2024-01-27T03:29:57.217 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-01-27T03:29:57.242 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-01-27T03:29:57.268 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-01-27T03:29:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-27T03:29:57.426 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-01-27T03:29:57.452 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 81 kB/s | 15 kB 00:00 2024-01-27T03:29:57.485 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): ceph-mgr-rook-19.0.0-926.g5ffe0641.el9. 104 kB/s | 52 kB 00:00 2024-01-27T03:29:57.505 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-27T03:29:57.536 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): python3-google-auth-2.26.2-1.el9.noarc 1.8 MB/s | 202 kB 00:00 2024-01-27T03:29:57.567 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-27T03:29:57.591 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 6/6 2024-01-27T03:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-01-27T03:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 442 kB/s | 59 kB 00:00 2024-01-27T03:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 831 kB/s | 90 kB 00:00 2024-01-27T03:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-01-27T03:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:29:57.761 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:29:57.761 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:29:58.281 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:29:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 6/6 2024-01-27T03:29:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/6 2024-01-27T03:29:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-27T03:29:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-27T03:29:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-27T03:29:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-27T03:29:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-27T03:29:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-27T03:29:58.569 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-27T03:29:58.606 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-27T03:29:58.647 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:58.729 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:29:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-27T03:29:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.26.2-1.el9.noarch 7/13 2024-01-27T03:29:58.890 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-27T03:29:59.032 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse 2024-01-27T03:29:59.603 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:24 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:29:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-27T03:29:59.688 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-27T03:29:59.721 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-27T03:29:59.756 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 885 k 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:29:59.757 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-01-27T03:29:59.758 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:29:59.758 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 964 k 2024-01-27T03:29:59.758 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.8 M 2024-01-27T03:29:59.758 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:29:59.770 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-27T03:29:59.792 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.0.0-926.g5ffe0641.el9.noarch 13/13 2024-01-27T03:29:59.905 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 535 kB/s | 79 kB 00:00 2024-01-27T03:30:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-926.g5ffe0641.el9.noarch 13/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.0.0-926.g5ffe0641.el9.noarch 1/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-27T03:30:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.26.2-1.el9.noarch 10/13 2024-01-27T03:30:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-27T03:30:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.26.2-1.el9.noarch 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-27T03:30:00.915 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-27T03:30:00.916 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-27T03:30:00.916 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-27T03:30:00.916 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-27T03:30:00.916 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-27T03:30:00.916 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-27T03:30:00.916 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-27T03:30:00.916 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:00.916 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:01.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-cephadm 2024-01-27T03:30:01.785 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:20 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:01.939 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:01.939 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 133 k 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:01.940 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-01-27T03:30:01.941 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-01-27T03:30:01.941 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:01.941 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 6.8 M 2024-01-27T03:30:01.941 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 33 M 2024-01-27T03:30:01.941 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:02.156 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2024-01-27T03:30:02.315 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-27T03:30:02.400 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.2 MB/s | 548 kB 00:00 2024-01-27T03:30:02.435 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): ceph-mgr-cephadm-19.0.0-926.g5ffe0641.el 300 kB/s | 133 kB 00:00 2024-01-27T03:30:02.449 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 431 kB/s | 58 kB 00:00 2024-01-27T03:30:02.466 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 762 kB/s | 51 kB 00:00 2024-01-27T03:30:02.467 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:02.468 INFO:teuthology.orchestra.run.smithi162.stdout:Total 13 MB/s | 6.8 MB 00:00 2024-01-27T03:30:02.468 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:30:02.478 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:30:02.478 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:30:02.567 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:30:02.568 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:30:02.871 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:30:03.033 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-27T03:30:03.085 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-27T03:30:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-27T03:30:03.696 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-27T03:30:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-27T03:30:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 6/6 2024-01-27T03:30:04.601 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 6/6 2024-01-27T03:30:04.601 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 1/6 2024-01-27T03:30:04.601 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-27T03:30:04.601 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-27T03:30:04.601 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-27T03:30:04.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:04.911 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:05.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2024-01-27T03:30:05.732 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:24 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:05.886 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:05.886 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:05.886 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:30:05.886 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:05.886 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 885 k 2024-01-27T03:30:05.887 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:30:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-27T03:30:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:05.887 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:05.887 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:05.887 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-01-27T03:30:05.887 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:05.888 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 964 k 2024-01-27T03:30:05.888 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.8 M 2024-01-27T03:30:05.888 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:06.176 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 272 kB/s | 79 kB 00:00 2024-01-27T03:30:06.219 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-fuse-19.0.0-926.g5ffe0641.el9.x86_6 2.6 MB/s | 885 kB 00:00 2024-01-27T03:30:06.220 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:06.220 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.8 MB/s | 964 kB 00:00 2024-01-27T03:30:06.220 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:30:06.241 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:30:06.241 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:30:06.315 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:30:06.315 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:30:06.497 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:30:06.637 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-27T03:30:06.709 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 2/2 2024-01-27T03:30:07.419 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 2/2 2024-01-27T03:30:07.419 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 1/2 2024-01-27T03:30:07.786 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-27T03:30:07.786 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:07.786 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:07.786 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-27T03:30:07.786 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:07.786 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:08.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-volume 2024-01-27T03:30:08.180 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-fuse-19.0.0-926.g5ffe0641.el9.x86_6 105 kB/s | 885 kB 00:08 2024-01-27T03:30:08.181 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:Total 114 kB/s | 964 kB 00:08 2024-01-27T03:30:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:30:08.205 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:30:08.206 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:30:08.285 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:30:08.285 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:30:08.493 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:30:08.621 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-27T03:30:08.637 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:27 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:08.687 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 2/2 2024-01-27T03:30:08.791 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:08.791 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:08.791 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-27T03:30:08.791 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:08.791 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:08.791 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 262 k 2024-01-27T03:30:08.791 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:08.792 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:08.792 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:08.792 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:30:08.792 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:08.792 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 262 k 2024-01-27T03:30:08.792 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.2 M 2024-01-27T03:30:08.792 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:09.374 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 2/2 2024-01-27T03:30:09.375 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 1/2 2024-01-27T03:30:09.415 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-volume-19.0.0-926.g5ffe0641.el9.noarch.rpm 421 kB/s | 262 kB 00:00 2024-01-27T03:30:09.416 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:09.416 INFO:teuthology.orchestra.run.smithi162.stdout:Total 420 kB/s | 262 kB 00:00 2024-01-27T03:30:09.416 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:30:09.424 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:30:09.424 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:30:09.459 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:30:09.460 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:30:09.669 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-27T03:30:09.669 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:09.669 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:30:09.669 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-27T03:30:09.669 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:09.669 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:09.678 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:30:09.742 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:30:09.758 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:30:09.759 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:30:09.759 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-27T03:30:09.759 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:09.932 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-volume 2024-01-27T03:30:10.520 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:35 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:10.679 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:19.0.0-926.g5ffe0641.el9 ceph-noarch 262 k 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:30:10.680 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:10.681 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 262 k 2024-01-27T03:30:10.681 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.2 M 2024-01-27T03:30:10.681 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:30:10.885 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:30:10.885 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:10.885 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:10.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:30:10.885 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:10.885 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:11.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2024-01-27T03:30:11.293 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-volume-19.0.0-926.g5ffe0641.el9.noarch.rpm 429 kB/s | 262 kB 00:00 2024-01-27T03:30:11.293 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:11.293 INFO:teuthology.orchestra.run.smithi110.stdout:Total 428 kB/s | 262 kB 00:00 2024-01-27T03:30:11.294 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:30:11.302 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:30:11.302 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:30:11.341 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:30:11.342 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:30:11.556 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:30:11.628 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:30:11.644 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:30:11.644 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:30:11.644 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-27T03:30:11.644 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:11.804 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:30 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:11.960 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 127 k 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 127 k 2024-01-27T03:30:11.961 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 456 k 2024-01-27T03:30:11.962 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:12.214 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-19.0.0-926.g5ffe0641.el9.x86_64. 503 kB/s | 127 kB 00:00 2024-01-27T03:30:12.214 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:12.214 INFO:teuthology.orchestra.run.smithi162.stdout:Total 500 kB/s | 127 kB 00:00 2024-01-27T03:30:12.215 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:30:12.220 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:30:12.220 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:30:12.277 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:30:12.277 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:30:12.434 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:30:12.537 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:12.669 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:19.0.0-926.g5ffe0641.el9.noarch 1/1 2024-01-27T03:30:12.669 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:12.670 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:30:12.670 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:19.0.0-926.g5ffe0641.el9.noarch 2024-01-27T03:30:12.670 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:12.670 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:13.019 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel 2024-01-27T03:30:13.032 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:13.342 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:13.342 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:13.342 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:13.342 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:13.342 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:13.343 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:13.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2024-01-27T03:30:13.614 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:38 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:13.766 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 127 k 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:30:13.767 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:13.768 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 127 k 2024-01-27T03:30:13.768 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 456 k 2024-01-27T03:30:13.768 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:30:14.074 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-19.0.0-926.g5ffe0641.el9.x86_64. 415 kB/s | 127 kB 00:00 2024-01-27T03:30:14.074 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:14.075 INFO:teuthology.orchestra.run.smithi110.stdout:Total 413 kB/s | 127 kB 00:00 2024-01-27T03:30:14.075 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:30:14.080 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:30:14.080 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:30:14.135 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:30:14.135 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:30:14.156 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:33 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:14.260 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:14.283 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:30:14.311 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:14.313 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:30:14.313 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:14.388 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:14.392 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2024-01-27T03:30:14.859 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:14.976 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:33 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:15.131 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:15.132 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:15.132 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:30:15.132 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:15.132 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:15.133 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 31 k 2024-01-27T03:30:15.133 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:15.133 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:15.133 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:15.133 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:30:15.133 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:15.133 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 31 k 2024-01-27T03:30:15.133 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 140 k 2024-01-27T03:30:15.133 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:15.151 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:15.151 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:15.152 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:30:15.152 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:15.152 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:15.152 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:15.377 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-19.0.0-926.g5ffe0641.el9.x86_64 128 kB/s | 31 kB 00:00 2024-01-27T03:30:15.377 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:15.377 INFO:teuthology.orchestra.run.smithi162.stdout:Total 127 kB/s | 31 kB 00:00 2024-01-27T03:30:15.377 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:30:15.381 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:30:15.381 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:30:15.412 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2 2024-01-27T03:30:15.413 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:30:15.413 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:30:15.504 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:30:15.616 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:15.993 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:15.998 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:40 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:16.104 INFO:teuthology.orchestra.run.smithi110.stdout:Package libcephfs2-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:16.157 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:16.158 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:30:16.158 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:16.232 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs-devel 2024-01-27T03:30:16.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:16.277 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:16.277 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:16.277 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:16.277 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:16.277 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:16.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2024-01-27T03:30:16.805 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:41 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:16.957 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:16.958 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:16.958 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:30:16.958 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:16.958 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:30:16.958 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 31 k 2024-01-27T03:30:16.958 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:16.958 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:30:16.958 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:16.958 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:30:16.959 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:16.959 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 31 k 2024-01-27T03:30:16.959 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 140 k 2024-01-27T03:30:16.959 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:30:17.043 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:36 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:17.128 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs-devel-19.0.0-926.g5ffe0641.el9.x86_64 184 kB/s | 31 kB 00:00 2024-01-27T03:30:17.129 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:17.129 INFO:teuthology.orchestra.run.smithi110.stdout:Total 183 kB/s | 31 kB 00:00 2024-01-27T03:30:17.129 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:30:17.132 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:30:17.132 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:30:17.144 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:17.161 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:30:17.162 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:30:17.195 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:17.196 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:30:17.196 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:17.251 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:30:17.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2024-01-27T03:30:17.350 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:17.741 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:17.849 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:36 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:17.950 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:18.002 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:18.003 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:30:18.003 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:18.009 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:18.009 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:18.009 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:30:18.009 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:18.009 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:18.009 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:18.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2024-01-27T03:30:18.211 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2 2024-01-27T03:30:18.648 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:37 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:18.748 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:18.793 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:43 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:18.799 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:18.801 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:30:18.801 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:18.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2024-01-27T03:30:18.894 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:18.945 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:18.946 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:30:18.946 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:19.025 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1 2024-01-27T03:30:19.449 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:38 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:19.550 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:19.600 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:44 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:19.601 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:19.602 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:30:19.603 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:19.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2024-01-27T03:30:19.703 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:19.754 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:19.755 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:30:19.755 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:19.835 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados 2024-01-27T03:30:20.248 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:39 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:20.349 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:20.400 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:20.401 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:30:20.401 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:20.447 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:45 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:20.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2024-01-27T03:30:20.548 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rados-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:20.599 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:20.600 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:30:20.600 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:20.688 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw 2024-01-27T03:30:21.049 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:40 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:21.152 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:21.204 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:21.205 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:30:21.205 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:21.264 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:46 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:21.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2024-01-27T03:30:21.364 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rgw-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:21.415 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:21.416 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:30:21.416 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:21.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs 2024-01-27T03:30:21.858 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:40 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:22.011 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:22.012 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:22.012 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-01-27T03:30:22.012 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:22.012 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:22.012 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 85 k 2024-01-27T03:30:22.012 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:22.012 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:22.012 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:22.012 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:30:22.013 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:22.013 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 85 k 2024-01-27T03:30:22.013 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 231 k 2024-01-27T03:30:22.013 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:22.071 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:47 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:22.173 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-cephfs-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:22.226 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:22.226 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:30:22.226 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:22.298 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd 2024-01-27T03:30:22.413 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-19.0.0-926.g5ffe0641.el9.x86_64.rpm 213 kB/s | 85 kB 00:00 2024-01-27T03:30:22.414 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:22.414 INFO:teuthology.orchestra.run.smithi162.stdout:Total 213 kB/s | 85 kB 00:00 2024-01-27T03:30:22.414 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:30:22.421 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:30:22.421 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:30:22.475 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:30:22.476 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:30:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:30:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:22.870 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:47 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:22.970 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rbd-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:30:23.021 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:23.022 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:30:23.022 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:23.102 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse 2024-01-27T03:30:23.149 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:23.451 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:23.451 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:23.451 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:23.451 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:23.451 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:23.451 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:23.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2024-01-27T03:30:23.690 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:48 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:23.854 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:23.855 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:23.855 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-01-27T03:30:23.855 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:23.855 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:30:23.855 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 85 k 2024-01-27T03:30:23.856 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:23.856 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:30:23.856 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:23.856 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:30:23.856 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:23.856 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 85 k 2024-01-27T03:30:23.856 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 231 k 2024-01-27T03:30:23.856 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:30:24.083 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-19.0.0-926.g5ffe0641.el9.x86_64.rpm 375 kB/s | 85 kB 00:00 2024-01-27T03:30:24.084 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:24.084 INFO:teuthology.orchestra.run.smithi110.stdout:Total 373 kB/s | 85 kB 00:00 2024-01-27T03:30:24.084 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:30:24.092 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:30:24.092 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:30:24.149 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:30:24.149 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:30:24.258 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:43 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:24.300 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:30:24.400 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:24.417 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:24.418 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:24.418 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:30:24.418 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:24.418 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:24.418 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 3.1 M 2024-01-27T03:30:24.418 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:24.419 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:24.419 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:24.419 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:30:24.419 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:24.419 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.1 M 2024-01-27T03:30:24.419 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 13 M 2024-01-27T03:30:24.419 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:24.808 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:25.091 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:25.091 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:25.091 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:30:25.091 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:25.091 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:25.091 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:25.367 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror 2024-01-27T03:30:25.950 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:50 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:26.050 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-19.0.0-926.g5ffe0641.el9.x86_64.rpm 1.9 MB/s | 3.1 MB 00:01 2024-01-27T03:30:26.051 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:26.051 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.9 MB/s | 3.1 MB 00:01 2024-01-27T03:30:26.051 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:30:26.062 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:30:26.062 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:30:26.111 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 3.1 M 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:30:26.112 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:26.113 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 3.1 M 2024-01-27T03:30:26.113 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 13 M 2024-01-27T03:30:26.113 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:30:26.143 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:30:26.143 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:30:26.582 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:30:26.651 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:26.675 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:26.675 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:30:26.675 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-27T03:30:26.675 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-27T03:30:26.675 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-27T03:30:26.675 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:26.771 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-19.0.0-926.g5ffe0641.el9.x86_64.rpm 4.7 MB/s | 3.1 MB 00:00 2024-01-27T03:30:26.771 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:26.772 INFO:teuthology.orchestra.run.smithi110.stdout:Total 4.7 MB/s | 3.1 MB 00:00 2024-01-27T03:30:26.772 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:30:26.781 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:30:26.781 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:30:26.858 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:30:26.858 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:30:27.290 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:30:27.354 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:27.379 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:27.379 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-27T03:30:27.379 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-27T03:30:27.379 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-27T03:30:27.379 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-27T03:30:27.379 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:27.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:27.897 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:27.897 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:27.897 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:27.898 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:27.898 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:28.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2024-01-27T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:28.712 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:47 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:28.781 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd 2024-01-27T03:30:28.865 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:28.866 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-01-27T03:30:28.866 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:28.866 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 170 k 2024-01-27T03:30:28.866 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:28.866 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:28.867 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:28.867 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:30:28.867 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:28.867 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 170 k 2024-01-27T03:30:28.867 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 494 k 2024-01-27T03:30:28.867 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:29.247 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-19.0.0-926.g5ffe0641.el9.x86_64.rpm 449 kB/s | 170 kB 00:00 2024-01-27T03:30:29.248 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:29.248 INFO:teuthology.orchestra.run.smithi162.stdout:Total 447 kB/s | 170 kB 00:00 2024-01-27T03:30:29.248 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:30:29.256 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:30:29.256 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:30:29.310 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:30:29.311 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:30:29.368 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:54 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:29.455 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:30:29.519 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:29.520 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:29.520 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-01-27T03:30:29.520 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 170 k 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 170 k 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 494 k 2024-01-27T03:30:29.521 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:30:29.571 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:29.750 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-19.0.0-926.g5ffe0641.el9.x86_64.rpm 746 kB/s | 170 kB 00:00 2024-01-27T03:30:29.751 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:29.751 INFO:teuthology.orchestra.run.smithi110.stdout:Total 741 kB/s | 170 kB 00:00 2024-01-27T03:30:29.751 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:30:29.758 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:30:29.759 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:30:29.814 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:30:29.814 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:30:29.961 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:30:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:30.048 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:30.331 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:30.331 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:30.331 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:30.505 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:30.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base-debuginfo 2024-01-27T03:30:30.817 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:30:30.817 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:30.818 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:30:30.818 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:30.818 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:30.818 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:31.115 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base-debuginfo 2024-01-27T03:30:31.146 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:50 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:31.308 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:31.308 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:30:31.308 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:31.308 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 67 M 2024-01-27T03:30:31.309 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-27T03:30:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 22 M 2024-01-27T03:30:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:31.309 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:31.309 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:31.309 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-01-27T03:30:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:31.309 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 89 M 2024-01-27T03:30:31.310 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 490 M 2024-01-27T03:30:31.310 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:31.704 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:56 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:31.862 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:31.862 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:31.862 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:30:31.862 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:31.862 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:30:31.862 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 67 M 2024-01-27T03:30:31.862 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-01-27T03:30:31.863 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 22 M 2024-01-27T03:30:31.863 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:31.863 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:30:31.863 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:31.863 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-01-27T03:30:31.863 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:31.863 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 89 M 2024-01-27T03:30:31.863 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 490 M 2024-01-27T03:30:31.863 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:30:34.059 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): ceph-debuginfo-19.0.0-926.g5ffe0641.el9. 8.1 MB/s | 22 MB 00:02 2024-01-27T03:30:37.679 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): ceph-debuginfo-19.0.0-926.g5ffe0641.el9. 3.8 MB/s | 22 MB 00:05 2024-01-27T03:30:37.867 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-base-debuginfo-19.0.0-926.g5ffe0641 10 MB/s | 67 MB 00:06 2024-01-27T03:30:37.869 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:37.869 INFO:teuthology.orchestra.run.smithi162.stdout:Total 14 MB/s | 89 MB 00:06 2024-01-27T03:30:37.870 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:30:37.874 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:30:37.874 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:30:38.458 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:30:38.458 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:30:41.570 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:30:42.012 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-base-debuginfo-19.0.0-926.g5ffe0641 6.6 MB/s | 67 MB 00:10 2024-01-27T03:30:42.014 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:30:42.014 INFO:teuthology.orchestra.run.smithi110.stdout:Total 8.8 MB/s | 89 MB 00:10 2024-01-27T03:30:42.014 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:30:42.020 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:30:42.020 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:30:42.597 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:30:42.597 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:30:45.680 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:30:48.596 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/2 2024-01-27T03:30:48.742 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 2/2 2024-01-27T03:30:52.158 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 2/2 2024-01-27T03:30:52.159 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/2 2024-01-27T03:30:52.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2/2 2024-01-27T03:30:52.432 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:52.432 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:30:52.432 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:52.432 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:52.433 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:52.433 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:30:52.636 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/2 2024-01-27T03:30:52.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-common-debuginfo 2024-01-27T03:30:52.713 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 2/2 2024-01-27T03:30:53.237 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:12 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:30:53.392 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:30:53.392 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:53.392 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:30:53.392 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:53.392 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:30:53.392 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 416 M 2024-01-27T03:30:53.392 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:53.392 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:30:53.392 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:30:53.393 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:30:53.393 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:30:53.393 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 416 M 2024-01-27T03:30:53.393 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.7 G 2024-01-27T03:30:53.393 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:30:56.169 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 2/2 2024-01-27T03:30:56.169 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/2 2024-01-27T03:30:56.437 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2/2 2024-01-27T03:30:56.438 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:56.438 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:30:56.438 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:56.438 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:30:56.438 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:56.438 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:30:56.672 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-common-debuginfo 2024-01-27T03:30:57.274 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:22 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:30:57.436 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 416 M 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:30:57.437 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:30:57.438 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 416 M 2024-01-27T03:30:57.438 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.7 G 2024-01-27T03:30:57.438 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:31:50.138 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-common-debuginfo-19.0.0-926.g5ffe0641.el9. 7.9 MB/s | 416 MB 00:52 2024-01-27T03:31:50.139 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:31:50.139 INFO:teuthology.orchestra.run.smithi110.stdout:Total 7.9 MB/s | 416 MB 00:52 2024-01-27T03:31:50.139 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:31:50.143 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:31:50.144 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:31:52.623 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:31:52.623 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:31:53.060 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-common-debuginfo-19.0.0-926.g5ffe0641.el9. 7.0 MB/s | 416 MB 00:59 2024-01-27T03:31:53.061 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:31:53.061 INFO:teuthology.orchestra.run.smithi162.stdout:Total 7.0 MB/s | 416 MB 00:59 2024-01-27T03:31:53.061 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:31:53.066 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:31:53.066 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:31:55.558 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:31:55.559 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:32:34.673 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:32:34.775 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:32:37.763 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:32:38.249 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:32:38.457 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:32:38.741 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:32:38.741 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:32:38.741 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:32:38.741 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:32:38.741 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:32:38.741 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:32:38.992 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-01-27T03:32:39.571 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:04 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:32:39.804 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 1.4 M 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.4 M 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 6.4 M 2024-01-27T03:32:39.805 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:32:40.488 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-92 1.9 MB/s | 1.4 MB 00:00 2024-01-27T03:32:40.489 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:32:40.489 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.9 MB/s | 1.4 MB 00:00 2024-01-27T03:32:40.489 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:32:40.493 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:32:40.493 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:32:40.534 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:32:40.534 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:32:40.787 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:32:40.878 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-926.g 1/1 2024-01-27T03:32:41.285 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-926.g 1/1 2024-01-27T03:32:41.557 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-926.g 1/1 2024-01-27T03:32:41.557 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:32:41.557 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:32:41.557 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:32:41.557 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:32:41.557 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:32:41.678 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:32:41.731 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw-debuginfo 2024-01-27T03:32:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:32:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:32:41.953 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:32:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:32:41.953 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:32:41.953 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:32:42.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-01-27T03:32:42.309 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:07 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:32:42.463 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:32:42.464 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:32:42.464 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:32:42.464 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:32:42.464 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:32:42.464 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 176 M 2024-01-27T03:32:42.464 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:32:42.465 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:32:42.465 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:32:42.465 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:32:42.465 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:32:42.465 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 176 M 2024-01-27T03:32:42.465 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 753 M 2024-01-27T03:32:42.465 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:32:42.777 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:04:01 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:32:42.944 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:32:42.944 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-01-27T03:32:42.944 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 1.4 M 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.4 M 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 6.4 M 2024-01-27T03:32:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:32:44.357 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-92 1.0 MB/s | 1.4 MB 00:01 2024-01-27T03:32:44.357 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:32:44.358 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.0 MB/s | 1.4 MB 00:01 2024-01-27T03:32:44.358 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:32:44.361 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:32:44.361 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:32:44.404 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:32:44.405 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:32:44.658 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:32:44.756 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-926.g 1/1 2024-01-27T03:32:45.190 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-926.g 1/1 2024-01-27T03:32:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-926.g 1/1 2024-01-27T03:32:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:32:45.450 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:32:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:32:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:32:45.450 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:32:45.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw-debuginfo 2024-01-27T03:32:46.184 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:04:05 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:32:46.337 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:32:46.338 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:32:46.338 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:32:46.338 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:32:46.338 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:32:46.339 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 176 M 2024-01-27T03:32:46.339 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:32:46.339 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:32:46.339 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:32:46.339 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:32:46.339 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:32:46.339 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 176 M 2024-01-27T03:32:46.339 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 753 M 2024-01-27T03:32:46.339 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:33:01.057 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-radosgw-debuginfo-19.0.0-926.g5ffe0641.el9 12 MB/s | 176 MB 00:14 2024-01-27T03:33:01.058 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:33:01.058 INFO:teuthology.orchestra.run.smithi162.stdout:Total 12 MB/s | 176 MB 00:14 2024-01-27T03:33:01.059 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:33:01.063 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:33:01.063 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:33:02.123 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:33:02.123 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:33:08.722 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-radosgw-debuginfo-19.0.0-926.g5ffe0641.el9 6.7 MB/s | 176 MB 00:26 2024-01-27T03:33:08.723 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:33:08.723 INFO:teuthology.orchestra.run.smithi110.stdout:Total 6.7 MB/s | 176 MB 00:26 2024-01-27T03:33:08.724 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:33:08.728 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:33:08.728 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:33:09.784 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:33:09.784 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:33:20.331 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:33:20.510 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x8 1/1 2024-01-27T03:33:25.186 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x8 1/1 2024-01-27T03:33:25.515 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x8 1/1 2024-01-27T03:33:25.516 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:33:25.516 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:33:25.516 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:33:25.516 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:33:25.516 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:33:25.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test-debuginfo 2024-01-27T03:33:26.262 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:04:45 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:33:26.424 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 672 M 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:33:26.425 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:33:26.426 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 672 M 2024-01-27T03:33:26.426 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 3.1 G 2024-01-27T03:33:26.426 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:33:27.848 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:33:28.011 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x8 1/1 2024-01-27T03:33:32.707 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x8 1/1 2024-01-27T03:33:33.033 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x8 1/1 2024-01-27T03:33:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:33:33.034 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:33:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:33:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:33:33.034 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:33:33.205 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test-debuginfo 2024-01-27T03:33:33.784 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:04:58 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:33:33.948 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:33:33.948 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:33:33.948 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 672 M 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 672 M 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 3.1 G 2024-01-27T03:33:33.949 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:36:21.625 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-test-debuginfo-19.0.0-926.g5ffe0641.el9.x8 3.8 MB/s | 672 MB 02:55 2024-01-27T03:36:21.626 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:36:21.626 INFO:teuthology.orchestra.run.smithi162.stdout:Total 3.8 MB/s | 672 MB 02:55 2024-01-27T03:36:21.626 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:36:21.631 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:36:21.631 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:36:22.014 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-test-debuginfo-19.0.0-926.g5ffe0641.el9.x8 4.0 MB/s | 672 MB 02:48 2024-01-27T03:36:22.015 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:36:22.015 INFO:teuthology.orchestra.run.smithi110.stdout:Total 4.0 MB/s | 672 MB 02:48 2024-01-27T03:36:22.015 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:36:22.020 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:36:22.020 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:36:25.589 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:36:25.589 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:36:26.002 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:36:26.002 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:37:36.636 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:37:36.839 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:37:37.024 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:37:37.246 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:37:46.311 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:37:46.615 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:37:46.615 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:37:46.615 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:37:46.616 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:37:46.616 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:37:46.616 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:37:46.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base-debuginfo 2024-01-27T03:37:46.924 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:37:47.193 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:37:47.193 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:37:47.193 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:37:47.193 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:37:47.193 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:37:47.193 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:37:47.479 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base-debuginfo 2024-01-27T03:37:47.486 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:09:06 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:37:47.591 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:37:47.642 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:37:47.643 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-27T03:37:47.643 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:37:47.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-debuginfo 2024-01-27T03:37:48.057 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:09:13 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:37:48.158 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 is already installed. 2024-01-27T03:37:48.209 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:37:48.210 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-01-27T03:37:48.210 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:37:48.285 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-debuginfo 2024-01-27T03:37:48.288 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:09:07 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:37:48.439 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:37:48.440 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:37:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:37:48.440 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:37:48.440 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:37:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 20 M 2024-01-27T03:37:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:37:48.441 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:37:48.441 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:37:48.441 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:37:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:37:48.441 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 20 M 2024-01-27T03:37:48.441 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 102 M 2024-01-27T03:37:48.441 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:37:48.863 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:09:13 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:37:49.015 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 20 M 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:37:49.016 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 20 M 2024-01-27T03:37:49.017 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 102 M 2024-01-27T03:37:49.017 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:37:50.206 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-mgr-debuginfo-19.0.0-926.g5ffe0641.el9.x86 11 MB/s | 20 MB 00:01 2024-01-27T03:37:50.206 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:37:50.207 INFO:teuthology.orchestra.run.smithi162.stdout:Total 11 MB/s | 20 MB 00:01 2024-01-27T03:37:50.207 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:37:50.210 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:37:50.210 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:37:50.358 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:37:50.358 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:37:51.645 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mgr-debuginfo-19.0.0-926.g5ffe0641.el9.x86 7.5 MB/s | 20 MB 00:02 2024-01-27T03:37:51.646 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:37:51.646 INFO:teuthology.orchestra.run.smithi110.stdout:Total 7.5 MB/s | 20 MB 00:02 2024-01-27T03:37:51.646 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:37:51.650 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:37:51.650 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:37:51.800 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:37:51.800 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:37:52.575 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:37:52.694 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:37:53.739 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:37:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:37:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:37:54.006 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:37:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:37:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:37:54.006 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:37:54.051 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:37:54.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mds-debuginfo 2024-01-27T03:37:54.168 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:37:54.739 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:09:13 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:37:54.892 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 48 M 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 48 M 2024-01-27T03:37:54.893 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 173 M 2024-01-27T03:37:54.894 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:37:55.153 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:37:55.408 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:37:55.408 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:37:55.409 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:37:55.409 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:37:55.409 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:37:55.409 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:37:55.604 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mds-debuginfo 2024-01-27T03:37:56.184 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:09:21 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:37:56.337 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:37:56.337 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:37:56.337 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:37:56.337 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 48 M 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 48 M 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 173 M 2024-01-27T03:37:56.338 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:38:11.138 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mds-debuginfo-19.0.0-926.g5ffe0641.el9.x86 3.2 MB/s | 48 MB 00:14 2024-01-27T03:38:11.139 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:38:11.139 INFO:teuthology.orchestra.run.smithi110.stdout:Total 3.2 MB/s | 48 MB 00:14 2024-01-27T03:38:11.139 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:38:11.143 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:38:11.143 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:38:11.455 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:38:11.455 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:38:16.177 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:38:16.277 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:38:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:38:17.922 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:38:17.922 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:38:17.923 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:38:17.923 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:38:17.923 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:38:17.923 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:38:18.129 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mon-debuginfo 2024-01-27T03:38:18.707 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:09:43 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:38:18.861 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 99 M 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:38:18.862 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:38:18.863 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 99 M 2024-01-27T03:38:18.863 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 449 M 2024-01-27T03:38:18.863 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:38:21.811 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-mds-debuginfo-19.0.0-926.g5ffe0641.el9.x86 1.8 MB/s | 48 MB 00:26 2024-01-27T03:38:21.812 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:38:21.812 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.8 MB/s | 48 MB 00:26 2024-01-27T03:38:21.813 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:38:21.817 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:38:21.817 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:38:22.131 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:38:22.132 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:38:26.873 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:38:26.992 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:38:28.496 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:38:28.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:38:28.772 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:38:28.772 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:38:28.772 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:38:28.772 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:38:28.772 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:38:28.967 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mon-debuginfo 2024-01-27T03:38:29.551 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:09:48 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:38:29.704 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 99 M 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 99 M 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 449 M 2024-01-27T03:38:29.705 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:38:35.313 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mon-debuginfo-19.0.0-926.g5ffe0641.el9.x86 6.0 MB/s | 99 MB 00:16 2024-01-27T03:38:35.314 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:38:35.315 INFO:teuthology.orchestra.run.smithi110.stdout:Total 6.0 MB/s | 99 MB 00:16 2024-01-27T03:38:35.315 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:38:35.319 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:38:35.319 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:38:35.929 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:38:35.929 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:38:46.201 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:38:46.294 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:38:49.243 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:38:49.507 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:38:49.507 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:38:49.507 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:38:49.507 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:38:49.507 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:38:49.508 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:38:49.693 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-osd-debuginfo 2024-01-27T03:38:50.303 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:10:15 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:38:50.458 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 286 M 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:38:50.459 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:38:50.460 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 286 M 2024-01-27T03:38:50.460 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.2 G 2024-01-27T03:38:50.460 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:38:56.783 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-mon-debuginfo-19.0.0-926.g5ffe0641.el9.x86 3.6 MB/s | 99 MB 00:27 2024-01-27T03:38:56.784 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:38:56.784 INFO:teuthology.orchestra.run.smithi162.stdout:Total 3.6 MB/s | 99 MB 00:27 2024-01-27T03:38:56.785 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:38:56.789 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:38:56.789 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:38:57.400 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:38:57.401 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:39:07.656 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:39:07.754 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:39:10.726 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:39:11.045 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:39:11.045 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:39:11.045 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:39:11.045 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:39:11.045 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:39:11.045 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:39:11.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-osd-debuginfo 2024-01-27T03:39:11.857 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:10:30 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:39:12.018 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 286 M 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 286 M 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.2 G 2024-01-27T03:39:12.019 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:39:49.810 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-osd-debuginfo-19.0.0-926.g5ffe0641.el9.x86 7.6 MB/s | 286 MB 00:37 2024-01-27T03:39:49.811 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:39:49.811 INFO:teuthology.orchestra.run.smithi162.stdout:Total 7.6 MB/s | 286 MB 00:37 2024-01-27T03:39:49.811 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:39:49.815 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:39:49.815 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:39:50.477 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-osd-debuginfo-19.0.0-926.g5ffe0641.el9.x86 4.8 MB/s | 286 MB 01:00 2024-01-27T03:39:50.477 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:39:50.478 INFO:teuthology.orchestra.run.smithi110.stdout:Total 4.8 MB/s | 286 MB 01:00 2024-01-27T03:39:50.478 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:39:50.483 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:39:50.483 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:39:51.520 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:39:51.520 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:39:52.187 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:39:52.187 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:40:20.770 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:40:20.862 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:40:21.468 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:40:21.573 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:40:28.470 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:40:28.738 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:40:28.739 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:28.739 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:40:28.739 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:28.739 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:28.739 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:40:28.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse-debuginfo 2024-01-27T03:40:29.089 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:40:29.362 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:40:29.362 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:29.362 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:40:29.362 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:29.362 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:29.362 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:40:29.532 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:11:48 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:40:29.582 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse-debuginfo 2024-01-27T03:40:29.686 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:40:29.686 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:29.687 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:40:29.687 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:29.687 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:40:29.687 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 5.9 M 2024-01-27T03:40:29.687 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:29.687 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:40:29.687 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:29.687 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:40:29.687 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:29.688 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 5.9 M 2024-01-27T03:40:29.688 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 28 M 2024-01-27T03:40:29.688 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:40:30.162 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:11:55 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:40:30.316 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 5.9 M 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 5.9 M 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 28 M 2024-01-27T03:40:30.317 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:40:30.832 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-fuse-debuginfo-19.0.0-926.g5ffe0641.el9.x8 5.2 MB/s | 5.9 MB 00:01 2024-01-27T03:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout:Total 5.2 MB/s | 5.9 MB 00:01 2024-01-27T03:40:30.833 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:40:30.836 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:40:30.836 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:40:30.904 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:40:30.904 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:40:31.469 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-fuse-debuginfo-19.0.0-926.g5ffe0641.el9.x8 5.2 MB/s | 5.9 MB 00:01 2024-01-27T03:40:31.470 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:31.470 INFO:teuthology.orchestra.run.smithi110.stdout:Total 5.1 MB/s | 5.9 MB 00:01 2024-01-27T03:40:31.470 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:40:31.474 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:40:31.474 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:40:31.540 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:40:31.541 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:40:31.642 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:40:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:32.272 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:40:32.326 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:32.375 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:32.603 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:32.603 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:32.603 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:40:32.603 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:32.603 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:32.603 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:40:32.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel-debuginfo 2024-01-27T03:40:32.967 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:33.247 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:33.247 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:33.247 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:40:33.247 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:33.248 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:33.248 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:40:33.377 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:11:52 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:40:33.451 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel-debuginfo 2024-01-27T03:40:33.532 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:40:33.533 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:33.533 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:40:33.533 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:33.533 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:40:33.533 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 108 k 2024-01-27T03:40:33.534 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:33.534 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:40:33.534 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:33.534 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:40:33.534 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:33.534 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 108 k 2024-01-27T03:40:33.534 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 439 k 2024-01-27T03:40:33.534 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:40:33.860 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-debuginfo-19.0.0-926.g5ffe0641.e 333 kB/s | 108 kB 00:00 2024-01-27T03:40:33.860 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:33.860 INFO:teuthology.orchestra.run.smithi162.stdout:Total 332 kB/s | 108 kB 00:00 2024-01-27T03:40:33.861 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:40:33.864 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:40:33.864 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:40:33.898 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:40:33.898 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:40:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:40:34.031 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:11:59 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:40:34.117 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:40:34.184 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:40:34.185 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:34.185 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:40:34.185 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 108 k 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 108 k 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 439 k 2024-01-27T03:40:34.186 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:40:34.397 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-debuginfo-19.0.0-926.g5ffe0641.e 515 kB/s | 108 kB 00:00 2024-01-27T03:40:34.398 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:34.398 INFO:teuthology.orchestra.run.smithi110.stdout:Total 512 kB/s | 108 kB 00:00 2024-01-27T03:40:34.398 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:40:34.401 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:40:34.401 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:40:34.435 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:40:34.435 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:40:34.535 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:40:34.552 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:40:34.669 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:40:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:40:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:34.812 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:40:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:34.812 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:40:34.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2-debuginfo 2024-01-27T03:40:35.043 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:40:35.310 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:40:35.310 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:35.311 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:40:35.311 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:35.311 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:35.311 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:40:35.520 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2-debuginfo 2024-01-27T03:40:35.583 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:11:54 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:40:35.737 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:40:35.738 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:35.738 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:40:35.738 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:35.738 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:40:35.738 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 5.3 M 2024-01-27T03:40:35.738 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:35.738 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:40:35.738 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:35.738 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:40:35.739 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:35.739 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 5.3 M 2024-01-27T03:40:35.739 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 25 M 2024-01-27T03:40:35.739 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:40:36.095 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:01 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:40:36.248 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:40:36.249 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:36.249 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:40:36.249 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:36.249 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:40:36.249 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 5.3 M 2024-01-27T03:40:36.249 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:36.249 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:40:36.249 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:36.249 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:40:36.250 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:36.250 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 5.3 M 2024-01-27T03:40:36.250 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 25 M 2024-01-27T03:40:36.250 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:40:36.372 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs2-debuginfo-19.0.0-926.g5ffe0641.el9.x 8.5 MB/s | 5.3 MB 00:00 2024-01-27T03:40:36.372 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:36.373 INFO:teuthology.orchestra.run.smithi162.stdout:Total 8.4 MB/s | 5.3 MB 00:00 2024-01-27T03:40:36.373 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:40:36.376 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:40:36.377 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:40:36.441 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:40:36.441 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:40:37.114 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:40:37.203 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:40:37.225 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs2-debuginfo-19.0.0-926.g5ffe0641.el9.x 5.5 MB/s | 5.3 MB 00:00 2024-01-27T03:40:37.226 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:37.227 INFO:teuthology.orchestra.run.smithi110.stdout:Total 5.5 MB/s | 5.3 MB 00:00 2024-01-27T03:40:37.227 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:40:37.230 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:40:37.230 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:40:37.294 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:40:37.294 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:40:37.762 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:40:37.956 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:40:38.057 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:40:38.058 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:38.058 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:40:38.058 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:38.058 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:38.058 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:40:38.074 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:40:38.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2-debuginfo 2024-01-27T03:40:38.638 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:40:38.817 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:11:57 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:40:38.928 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:40:38.928 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:38.928 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:40:38.928 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:38.928 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:38.929 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:40:38.969 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 44 M 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:40:38.970 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 44 M 2024-01-27T03:40:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 252 M 2024-01-27T03:40:38.971 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:40:39.127 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2-debuginfo 2024-01-27T03:40:39.703 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:04 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:40:39.855 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:40:39.856 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:39.856 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:40:39.856 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:39.856 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:40:39.857 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 44 M 2024-01-27T03:40:39.857 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:39.857 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:40:39.857 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:39.857 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:40:39.857 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:39.857 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 44 M 2024-01-27T03:40:39.857 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 252 M 2024-01-27T03:40:39.857 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:40:42.791 INFO:teuthology.orchestra.run.smithi162.stdout:librados2-debuginfo-19.0.0-926.g5ffe0641.el9.x8 12 MB/s | 44 MB 00:03 2024-01-27T03:40:42.792 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:42.792 INFO:teuthology.orchestra.run.smithi162.stdout:Total 12 MB/s | 44 MB 00:03 2024-01-27T03:40:42.792 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:40:42.796 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:40:42.796 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:40:43.100 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:40:43.100 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:40:48.086 INFO:teuthology.orchestra.run.smithi110.stdout:librados2-debuginfo-19.0.0-926.g5ffe0641.el9.x8 5.4 MB/s | 44 MB 00:08 2024-01-27T03:40:48.087 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:48.088 INFO:teuthology.orchestra.run.smithi110.stdout:Total 5.4 MB/s | 44 MB 00:08 2024-01-27T03:40:48.088 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:40:48.092 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:40:48.092 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:40:48.157 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:40:48.263 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:48.392 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:40:48.392 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:40:50.104 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:50.390 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:50.390 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:50.390 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:40:50.391 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:50.391 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:50.391 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:40:50.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1-debuginfo 2024-01-27T03:40:51.162 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:12:10 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:40:51.315 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 52 M 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:40:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:40:51.317 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 52 M 2024-01-27T03:40:51.317 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 308 M 2024-01-27T03:40:51.317 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:40:53.404 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:40:53.501 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:54.209 INFO:teuthology.orchestra.run.smithi162.stdout:librbd1-debuginfo-19.0.0-926.g5ffe0641.el9.x86_ 18 MB/s | 52 MB 00:02 2024-01-27T03:40:54.210 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:54.211 INFO:teuthology.orchestra.run.smithi162.stdout:Total 18 MB/s | 52 MB 00:02 2024-01-27T03:40:54.211 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:40:54.215 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:40:54.215 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:40:54.569 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:40:54.569 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:40:55.419 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:55.689 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_6 1/1 2024-01-27T03:40:55.689 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:55.689 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:40:55.689 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:40:55.689 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:55.689 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:40:55.929 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1-debuginfo 2024-01-27T03:40:56.506 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:21 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:40:56.657 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 52 M 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:40:56.658 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:40:56.659 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 52 M 2024-01-27T03:40:56.659 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 308 M 2024-01-27T03:40:56.659 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:40:59.256 INFO:teuthology.orchestra.run.smithi110.stdout:librbd1-debuginfo-19.0.0-926.g5ffe0641.el9.x86_ 20 MB/s | 52 MB 00:02 2024-01-27T03:40:59.257 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:40:59.257 INFO:teuthology.orchestra.run.smithi110.stdout:Total 20 MB/s | 52 MB 00:02 2024-01-27T03:40:59.257 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:40:59.261 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:40:59.261 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:40:59.608 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:40:59.608 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:41:00.604 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:41:00.724 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librbd1-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:02.932 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:03.247 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:03.247 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:03.248 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:41:03.248 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:03.248 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:03.248 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:41:03.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs-debuginfo 2024-01-27T03:41:04.033 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:12:23 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:41:04.185 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:41:04.186 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:04.186 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:41:04.186 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:04.186 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:41:04.186 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 615 k 2024-01-27T03:41:04.187 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:04.187 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:41:04.187 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:04.187 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:41:04.187 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:04.187 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 615 k 2024-01-27T03:41:04.187 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.1 M 2024-01-27T03:41:04.187 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:41:04.628 INFO:teuthology.orchestra.run.smithi162.stdout:python3-cephfs-debuginfo-19.0.0-926.g5ffe0641.e 1.4 MB/s | 615 kB 00:00 2024-01-27T03:41:04.629 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:04.629 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.4 MB/s | 615 kB 00:00 2024-01-27T03:41:04.629 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:41:04.632 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:41:04.632 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:41:04.669 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:41:04.669 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:41:04.838 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:41:04.960 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:41:05.352 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:41:05.599 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:41:05.627 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:41:05.627 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:05.627 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:41:05.627 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:05.627 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:05.627 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:41:05.696 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librbd1-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:05.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados-debuginfo 2024-01-27T03:41:06.393 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:12:25 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:41:06.546 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:41:06.547 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:06.547 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 770 k 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 770 k 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.7 M 2024-01-27T03:41:06.548 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:41:07.098 INFO:teuthology.orchestra.run.smithi162.stdout:python3-rados-debuginfo-19.0.0-926.g5ffe0641.el 1.4 MB/s | 770 kB 00:00 2024-01-27T03:41:07.099 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:07.099 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.4 MB/s | 770 kB 00:00 2024-01-27T03:41:07.099 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:41:07.102 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:41:07.103 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:41:07.140 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:41:07.141 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:41:07.322 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:41:07.423 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-debuginfo-2:19.0.0-926.g5ffe0641.el9.x 1/1 2024-01-27T03:41:07.865 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-926.g5ffe0641.el9.x 1/1 2024-01-27T03:41:07.965 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:08.143 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-926.g5ffe0641.el9.x 1/1 2024-01-27T03:41:08.143 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:08.143 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:41:08.143 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:08.143 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:08.143 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:41:08.250 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:08.250 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:08.250 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:41:08.250 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:08.250 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:08.250 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:41:08.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd-debuginfo 2024-01-27T03:41:08.473 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs-debuginfo 2024-01-27T03:41:08.929 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:12:27 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:41:09.052 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:34 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:41:09.085 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:41:09.086 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:09.086 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:41:09.086 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:09.086 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:41:09.086 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 1.1 M 2024-01-27T03:41:09.086 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:09.086 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:41:09.086 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:09.086 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:41:09.087 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:09.087 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.1 M 2024-01-27T03:41:09.087 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 4.3 M 2024-01-27T03:41:09.087 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:41:09.204 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:41:09.204 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 615 k 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 615 k 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.1 M 2024-01-27T03:41:09.205 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:41:10.067 INFO:teuthology.orchestra.run.smithi110.stdout:python3-cephfs-debuginfo-19.0.0-926.g5ffe0641.e 714 kB/s | 615 kB 00:00 2024-01-27T03:41:10.067 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:10.068 INFO:teuthology.orchestra.run.smithi110.stdout:Total 713 kB/s | 615 kB 00:00 2024-01-27T03:41:10.068 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:41:10.071 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:41:10.071 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:41:10.109 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:41:10.109 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:41:10.168 INFO:teuthology.orchestra.run.smithi162.stdout:python3-rbd-debuginfo-19.0.0-926.g5ffe0641.el9. 1.0 MB/s | 1.1 MB 00:01 2024-01-27T03:41:10.168 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:10.169 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.0 MB/s | 1.1 MB 00:01 2024-01-27T03:41:10.169 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:41:10.173 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:41:10.173 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:41:10.217 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:41:10.217 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:41:10.272 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:41:10.374 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:41:10.463 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:41:10.550 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:10.819 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:41:10.961 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:11.082 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-926.g5ffe0641.el9. 1/1 2024-01-27T03:41:11.082 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:11.082 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:41:11.082 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:11.082 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:11.082 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:41:11.256 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:11.257 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:41:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:11.257 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:41:11.306 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados-debuginfo 2024-01-27T03:41:11.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw-debuginfo 2024-01-27T03:41:11.880 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:36 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:41:12.025 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:12:31 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:41:12.031 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:41:12.031 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:12.032 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:41:12.032 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:12.032 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:41:12.032 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 770 k 2024-01-27T03:41:12.032 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:12.032 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:41:12.032 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:12.032 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:41:12.032 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:12.033 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 770 k 2024-01-27T03:41:12.033 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.7 M 2024-01-27T03:41:12.033 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:41:12.180 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 337 k 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:41:12.181 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:12.182 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 337 k 2024-01-27T03:41:12.182 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.0 M 2024-01-27T03:41:12.182 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:41:12.462 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rados-debuginfo-19.0.0-926.g5ffe0641.el 1.8 MB/s | 770 kB 00:00 2024-01-27T03:41:12.463 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:12.463 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.7 MB/s | 770 kB 00:00 2024-01-27T03:41:12.463 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:41:12.466 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:41:12.467 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:41:12.504 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:41:12.505 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:41:12.571 INFO:teuthology.orchestra.run.smithi162.stdout:python3-rgw-debuginfo-19.0.0-926.g5ffe0641.el9. 866 kB/s | 337 kB 00:00 2024-01-27T03:41:12.571 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:12.572 INFO:teuthology.orchestra.run.smithi162.stdout:Total 863 kB/s | 337 kB 00:00 2024-01-27T03:41:12.572 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:41:12.575 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:41:12.575 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:41:12.611 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:41:12.611 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:41:12.678 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:41:12.743 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:41:12.785 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-debuginfo-2:19.0.0-926.g5ffe0641.el9.x 1/1 2024-01-27T03:41:12.836 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:13.220 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-926.g5ffe0641.el9.x 1/1 2024-01-27T03:41:13.498 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-926.g5ffe0641.el9.x 1/1 2024-01-27T03:41:13.498 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:13.498 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:41:13.498 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:13.498 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:13.498 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:41:13.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:13.501 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:13.501 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:41:13.501 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:13.501 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:13.501 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:41:13.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse-debuginfo 2024-01-27T03:41:13.753 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd-debuginfo 2024-01-27T03:41:14.257 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:12:33 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:41:14.330 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:39 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:41:14.410 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:41:14.411 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:41:14.411 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:14.411 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:41:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 509 k 2024-01-27T03:41:14.411 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:14.411 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:41:14.412 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:14.412 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:41:14.412 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:14.412 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 509 k 2024-01-27T03:41:14.412 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.5 M 2024-01-27T03:41:14.412 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:41:14.482 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:41:14.483 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:14.483 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:41:14.483 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:14.483 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:41:14.483 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 1.1 M 2024-01-27T03:41:14.483 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:14.484 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:41:14.484 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:14.484 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:41:14.484 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:14.484 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.1 M 2024-01-27T03:41:14.484 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 4.3 M 2024-01-27T03:41:14.484 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:41:14.958 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-debuginfo-19.0.0-926.g5ffe0641.el9.x86 933 kB/s | 509 kB 00:00 2024-01-27T03:41:14.958 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:14.958 INFO:teuthology.orchestra.run.smithi162.stdout:Total 931 kB/s | 509 kB 00:00 2024-01-27T03:41:14.959 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:41:14.962 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:41:14.962 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:41:14.999 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:41:15.000 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:41:15.159 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:41:15.274 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:15.440 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rbd-debuginfo-19.0.0-926.g5ffe0641.el9. 1.2 MB/s | 1.1 MB 00:00 2024-01-27T03:41:15.441 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:15.441 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.2 MB/s | 1.1 MB 00:00 2024-01-27T03:41:15.441 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:41:15.445 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:41:15.445 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:41:15.487 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:41:15.487 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:41:15.661 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:15.702 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:41:15.798 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:15.944 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:15.944 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:15.944 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:41:15.944 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:15.944 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:15.944 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:41:16.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror-debuginfo 2024-01-27T03:41:16.238 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:16.526 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:16.527 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:16.527 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:41:16.527 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:16.527 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:16.527 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:41:16.735 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:12:35 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:41:16.748 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw-debuginfo 2024-01-27T03:41:16.888 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:41:16.889 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:16.889 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:41:16.889 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:16.889 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:41:16.890 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 52 M 2024-01-27T03:41:16.890 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:16.890 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:41:16.890 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:16.890 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:41:16.890 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:16.890 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 52 M 2024-01-27T03:41:16.890 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 298 M 2024-01-27T03:41:16.890 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:41:17.329 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:42 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:41:17.480 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:41:17.481 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:17.481 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:41:17.481 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 337 k 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 337 k 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.0 M 2024-01-27T03:41:17.482 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:41:18.102 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rgw-debuginfo-19.0.0-926.g5ffe0641.el9. 544 kB/s | 337 kB 00:00 2024-01-27T03:41:18.102 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:18.103 INFO:teuthology.orchestra.run.smithi110.stdout:Total 543 kB/s | 337 kB 00:00 2024-01-27T03:41:18.103 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:41:18.106 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:41:18.106 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:41:18.143 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:41:18.143 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:41:18.280 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:41:18.393 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:18.794 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:19.078 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86 1/1 2024-01-27T03:41:19.078 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:19.078 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:41:19.079 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:19.079 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:19.079 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:41:19.287 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse-debuginfo 2024-01-27T03:41:19.852 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-debuginfo-19.0.0-926.g5ffe0641.el9.x 17 MB/s | 52 MB 00:02 2024-01-27T03:41:19.853 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:19.853 INFO:teuthology.orchestra.run.smithi162.stdout:Total 17 MB/s | 52 MB 00:02 2024-01-27T03:41:19.854 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:41:19.858 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:41:19.858 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:41:19.864 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:44 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:41:20.016 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 509 k 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:41:20.017 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:20.018 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 509 k 2024-01-27T03:41:20.018 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.5 M 2024-01-27T03:41:20.018 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:41:20.194 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:41:20.195 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:41:20.997 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-debuginfo-19.0.0-926.g5ffe0641.el9.x86 520 kB/s | 509 kB 00:00 2024-01-27T03:41:20.997 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:20.998 INFO:teuthology.orchestra.run.smithi110.stdout:Total 519 kB/s | 509 kB 00:00 2024-01-27T03:41:20.998 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:41:21.002 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:41:21.002 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:41:21.043 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:41:21.043 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:41:21.204 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:41:21.312 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:21.702 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:21.987 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:21.987 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:21.987 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:41:21.987 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:21.987 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:21.987 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:41:22.187 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror-debuginfo 2024-01-27T03:41:22.767 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:47 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:41:22.921 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:41:22.921 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 52 M 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 52 M 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 298 M 2024-01-27T03:41:22.922 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:41:26.114 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:41:26.225 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:41:28.389 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:41:28.399 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-debuginfo-19.0.0-926.g5ffe0641.el9.x 9.4 MB/s | 52 MB 00:05 2024-01-27T03:41:28.401 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:28.401 INFO:teuthology.orchestra.run.smithi110.stdout:Total 9.4 MB/s | 52 MB 00:05 2024-01-27T03:41:28.401 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:41:28.405 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:41:28.405 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:41:28.676 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:41:28.676 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:28.676 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:41:28.676 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:28.677 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:28.677 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:41:28.740 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:41:28.740 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:41:28.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd-debuginfo 2024-01-27T03:41:29.437 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:12:48 ago on Sat 27 Jan 2024 03:28:41 AM UTC. 2024-01-27T03:41:29.591 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 1.3 M 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.3 M 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 5.5 M 2024-01-27T03:41:29.592 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-27T03:41:29.912 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-debuginfo-19.0.0-926.g5ffe0641.el9.x86_ 3.9 MB/s | 1.3 MB 00:00 2024-01-27T03:41:29.912 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:29.913 INFO:teuthology.orchestra.run.smithi162.stdout:Total 3.9 MB/s | 1.3 MB 00:00 2024-01-27T03:41:29.913 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-27T03:41:29.916 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-27T03:41:29.916 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-27T03:41:29.956 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-27T03:41:29.957 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-27T03:41:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-27T03:41:30.303 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:30.777 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:31.054 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:31.054 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:31.054 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-27T03:41:31.054 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:31.054 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:31.054 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-27T03:41:31.270 DEBUG:teuthology.parallel:result is None 2024-01-27T03:41:34.626 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:41:34.739 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:41:36.834 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:41:37.119 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_ 1/1 2024-01-27T03:41:37.119 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:37.119 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:41:37.119 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:37.119 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:37.120 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:41:37.316 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd-debuginfo 2024-01-27T03:41:37.894 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:13:02 ago on Sat 27 Jan 2024 03:28:35 AM UTC. 2024-01-27T03:41:38.048 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-01-27T03:41:38.048 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:38.048 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-01-27T03:41:38.048 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:38.048 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-01-27T03:41:38.048 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-926.g5ffe0641.el9 ceph 1.3 M 2024-01-27T03:41:38.049 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:38.049 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-01-27T03:41:38.049 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-01-27T03:41:38.049 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-01-27T03:41:38.049 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:38.049 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.3 M 2024-01-27T03:41:38.049 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 5.5 M 2024-01-27T03:41:38.049 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-01-27T03:41:38.519 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-debuginfo-19.0.0-926.g5ffe0641.el9.x86_ 2.7 MB/s | 1.3 MB 00:00 2024-01-27T03:41:38.519 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-01-27T03:41:38.520 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.7 MB/s | 1.3 MB 00:00 2024-01-27T03:41:38.520 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-01-27T03:41:38.523 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-01-27T03:41:38.523 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-01-27T03:41:38.563 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-01-27T03:41:38.563 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-01-27T03:41:38.798 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-01-27T03:41:38.901 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:39.351 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:39.617 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 1/1 2024-01-27T03:41:39.617 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:39.617 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-01-27T03:41:39.617 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-debuginfo-2:19.0.0-926.g5ffe0641.el9.x86_64 2024-01-27T03:41:39.617 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:39.617 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-01-27T03:41:39.809 DEBUG:teuthology.parallel:result is None 2024-01-27T03:41:39.810 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ffe0641f18e6ec692535294010d9456630e7923 2024-01-27T03:41:40.043 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-27T03:41:40.063 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-926.g5ffe0641.el9 2024-01-27T03:41:40.064 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-926.g5ffe0641.el9 2024-01-27T03:41:40.064 INFO:teuthology.task.install:The correct ceph version 19.0.0-926.g5ffe0641 is installed. 2024-01-27T03:41:40.066 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ffe0641f18e6ec692535294010d9456630e7923 2024-01-27T03:41:40.208 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-27T03:41:40.232 INFO:teuthology.orchestra.run.smithi162.stdout:19.0.0-926.g5ffe0641.el9 2024-01-27T03:41:40.233 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-926.g5ffe0641.el9 2024-01-27T03:41:40.233 INFO:teuthology.task.install:The correct ceph version 19.0.0-926.g5ffe0641 is installed. 2024-01-27T03:41:40.234 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-27T03:41:40.234 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:40.235 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-27T03:41:40.261 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:41:40.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-27T03:41:40.305 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-27T03:41:40.305 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:40.305 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-01-27T03:41:40.332 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-27T03:41:40.398 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:41:40.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-01-27T03:41:40.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-27T03:41:40.500 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-27T03:41:40.500 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:40.501 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-27T03:41:40.527 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-27T03:41:40.594 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:41:40.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-27T03:41:40.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-27T03:41:40.692 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-27T03:41:40.692 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:40.693 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-01-27T03:41:40.719 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-27T03:41:40.786 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:41:40.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-01-27T03:41:40.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-27T03:41:40.886 INFO:teuthology.run_tasks:Running task ceph... 2024-01-27T03:41:40.969 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-27T03:41:40.969 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2024-01-27T03:41:40.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /var/log/ceph 2024-01-27T03:41:40.999 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-27T03:41:40.999 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-27T03:41:41.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-27T03:41:41.065 INFO:tasks.ceph:Creating extra log directories... 2024-01-27T03:41:41.065 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-27T03:41:41.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-27T03:41:41.142 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-27T03:41:41.142 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-27T03:41:41.142 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300', 'branch': 'wip-read-balancer-mgr-module', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7535300', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-read-balancer-mgr-module'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '5ffe0641f18e6ec692535294010d9456630e7923', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5ffe0641f18e6ec692535294010d9456630e7923'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-read-balancer-mgr-module', 'sha1': '5ffe0641f18e6ec692535294010d9456630e7923'}}, 'owner': 'scheduled_lflores@teuthology', 'priority': 74, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 877, 'sha1': '5ffe0641f18e6ec692535294010d9456630e7923', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-read-balancer-mgr-module', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '5ffe0641f18e6ec692535294010d9456630e7923', 'targets': {'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE7jbvzpRRcKWxC8MKwEp6/2Eg7/ebLoAtLx3fCHrCP4j6YgaKbk2gzjCrCnqBSa8R7LP8NJKUc2BF0mFs8jIH8=', 'smithi162.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJpZBxOlpfYmIDdeeZrTsodR/7nDNca1GqxhvsdenHo1pug6H89ZquNR7aY5aBc19vkT9OGARRG+poLeF0c8MI4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c856e0f9244dd29a50697123b480d63f16a4ed81', 'timestamp': '2024-01-27_03:09:32', 'tube': 'smithi', 'user': 'lflores', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-27T03:41:41.143 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-27T03:41:41.172 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-27T03:41:41.186 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2024-01-27T03:41:41.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m0777 -- /var/run/ceph 2024-01-27T03:41:41.254 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:41.254 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-01-27T03:41:41.307 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-27T03:41:41.307 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-01-27T03:41:41.361 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-27T03:41:41.361 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-27T03:41:41.361 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-01-27T03:41:41.361 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-27T03:41:41.361 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-01-27T03:41:41.361 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-01-27 03:30:27.623113692 +0000 2024-01-27T03:41:41.361 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-01-27 03:27:50.154802096 +0000 2024-01-27T03:41:41.362 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-01-27 03:27:50.154802096 +0000 2024-01-27T03:41:41.362 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-01-27 03:27:50.154802096 +0000 2024-01-27T03:41:41.362 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-27T03:41:41.427 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-01-27T03:41:41.427 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-01-27T03:41:41.427 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.6965e-05 s, 5.3 MB/s 2024-01-27T03:41:41.428 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-27T03:41:41.484 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-01-27T03:41:41.539 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-27T03:41:41.540 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-27T03:41:41.540 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-01-27T03:41:41.540 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-27T03:41:41.540 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-01-27T03:41:41.540 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-01-27 03:30:27.624113675 +0000 2024-01-27T03:41:41.540 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-01-27 03:27:50.476796649 +0000 2024-01-27T03:41:41.540 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-01-27 03:27:50.476796649 +0000 2024-01-27T03:41:41.540 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-01-27 03:27:50.476796649 +0000 2024-01-27T03:41:41.540 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-27T03:41:41.604 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-01-27T03:41:41.604 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-01-27T03:41:41.604 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.6787e-05 s, 5.3 MB/s 2024-01-27T03:41:41.605 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-27T03:41:41.661 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-01-27T03:41:41.715 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-27T03:41:41.715 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-27T03:41:41.715 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-01-27T03:41:41.715 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-27T03:41:41.715 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-01-27T03:41:41.716 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-01-27 03:30:27.624113675 +0000 2024-01-27T03:41:41.716 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-01-27 03:27:50.795791252 +0000 2024-01-27T03:41:41.716 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-01-27 03:27:50.795791252 +0000 2024-01-27T03:41:41.716 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-01-27 03:27:50.795791252 +0000 2024-01-27T03:41:41.716 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-27T03:41:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-01-27T03:41:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-01-27T03:41:41.782 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 7.3131e-05 s, 7.0 MB/s 2024-01-27T03:41:41.782 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-27T03:41:41.838 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-01-27T03:41:41.893 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-27T03:41:41.893 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-27T03:41:41.893 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-01-27T03:41:41.893 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-27T03:41:41.893 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-01-27T03:41:41.893 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-01-27 03:30:27.625113659 +0000 2024-01-27T03:41:41.893 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-01-27 03:27:51.125785670 +0000 2024-01-27T03:41:41.894 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-01-27 03:27:51.125785670 +0000 2024-01-27T03:41:41.894 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-01-27 03:27:51.125785670 +0000 2024-01-27T03:41:41.894 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-27T03:41:41.957 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-01-27T03:41:41.958 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-01-27T03:41:41.958 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000102831 s, 5.0 MB/s 2024-01-27T03:41:41.959 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-27T03:41:42.014 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:42.014 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:41:42.014 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-01-27T03:41:42.031 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-27T03:41:42.031 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-01-27T03:41:42.087 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-27T03:41:42.087 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-27T03:41:42.087 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-01-27T03:41:42.087 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-27T03:41:42.088 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-01-27T03:41:42.088 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-27 03:30:26.920161033 +0000 2024-01-27T03:41:42.088 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-27 03:27:50.171639167 +0000 2024-01-27T03:41:42.088 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-27 03:27:50.171639167 +0000 2024-01-27T03:41:42.088 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-01-27 03:27:50.171639167 +0000 2024-01-27T03:41:42.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-27T03:41:42.154 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-27T03:41:42.154 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-27T03:41:42.154 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000214675 s, 2.4 MB/s 2024-01-27T03:41:42.156 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-27T03:41:42.213 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-01-27T03:41:42.268 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-27T03:41:42.268 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-27T03:41:42.268 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-01-27T03:41:42.269 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-27T03:41:42.269 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-01-27T03:41:42.269 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-27 03:30:26.921161013 +0000 2024-01-27T03:41:42.269 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-27 03:27:50.529632053 +0000 2024-01-27T03:41:42.269 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-27 03:27:50.529632053 +0000 2024-01-27T03:41:42.269 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-01-27 03:27:50.529632053 +0000 2024-01-27T03:41:42.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-27T03:41:42.334 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-27T03:41:42.334 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-27T03:41:42.335 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.00022012 s, 2.3 MB/s 2024-01-27T03:41:42.336 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-27T03:41:42.393 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-01-27T03:41:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-27T03:41:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-27T03:41:42.449 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-01-27T03:41:42.449 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-27T03:41:42.449 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-01-27T03:41:42.449 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-27 03:30:26.921161013 +0000 2024-01-27T03:41:42.449 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-27 03:27:50.880625076 +0000 2024-01-27T03:41:42.449 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-27 03:27:50.880625076 +0000 2024-01-27T03:41:42.449 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-01-27 03:27:50.880625076 +0000 2024-01-27T03:41:42.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-27T03:41:42.515 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-27T03:41:42.515 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-27T03:41:42.515 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000209093 s, 2.4 MB/s 2024-01-27T03:41:42.517 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-27T03:41:42.573 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-01-27T03:41:42.628 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-27T03:41:42.628 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-27T03:41:42.628 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-01-27T03:41:42.629 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-27T03:41:42.629 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-01-27T03:41:42.629 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-27 03:30:26.922160994 +0000 2024-01-27T03:41:42.629 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-27 03:27:51.233618061 +0000 2024-01-27T03:41:42.629 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-27 03:27:51.233618061 +0000 2024-01-27T03:41:42.629 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-01-27 03:27:51.233618061 +0000 2024-01-27T03:41:42.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-27T03:41:42.694 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-27T03:41:42.694 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-27T03:41:42.694 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000201129 s, 2.5 MB/s 2024-01-27T03:41:42.695 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-27T03:41:42.750 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:42.751 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi162.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-27T03:41:42.751 INFO:tasks.ceph:Generating config... 2024-01-27T03:41:42.752 INFO:tasks.ceph:[client] debug monc = 20 2024-01-27T03:41:42.752 INFO:tasks.ceph:[client] debug ms = 1 2024-01-27T03:41:42.752 INFO:tasks.ceph:[client] debug objecter = 20 2024-01-27T03:41:42.752 INFO:tasks.ceph:[client] debug rados = 20 2024-01-27T03:41:42.752 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-01-27T03:41:42.752 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-01-27T03:41:42.752 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-01-27T03:41:42.752 INFO:tasks.ceph:[global] ms type = async 2024-01-27T03:41:42.752 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-01-27T03:41:42.752 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-01-27T03:41:42.753 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-01-27T03:41:42.753 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-27T03:41:42.753 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] osd class default list = * 2024-01-27T03:41:42.754 INFO:tasks.ceph:[osd] osd class load list = * 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-01-27T03:41:42.755 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-27T03:41:42.755 INFO:tasks.ceph:Setting up mon.a... 2024-01-27T03:41:42.755 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-27T03:41:42.814 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.keyring 2024-01-27T03:41:42.817 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-27T03:41:42.864 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-27T03:41:42.891 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.110'), ('mon.c', '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]'), ('mon.b', '172.21.15.162')] 2024-01-27T03:41:42.891 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.162', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-01-27T03:41:42.892 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:42.892 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-27T03:41:42.947 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.110 --addv c '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]' --add b 172.21.15.162 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:41:43.031 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:41:43.031 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:setting min_mon_release = quincy 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 17 (quincy) 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:election_strategy: 1 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:41:43.032 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-27T03:41:43.033 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-27T03:41:43.047 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 23f901d8-28a2-405b-9f8e-3221d4911c2c... 2024-01-27T03:41:43.048 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-27T03:41:43.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2024-01-27T03:41:43.138 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2024-01-27T03:41:43.139 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2024-01-27T03:41:43.140 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.162" 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: mon election default strategy = 1 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 5000 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: ms type = async 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: osd heartbeat grace = 80 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2024-01-27T03:41:43.141 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug trim objects = true 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2024-01-27T03:41:43.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: bdev async discard = True 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: bdev enable discard = True 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore allocator = bitmap 2024-01-27T03:41:43.143 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore zero block detection = True 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 1 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 1 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 10 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: osd class default list = * 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: osd class load list = * 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: osd client watch timeout = 120 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify cached snaps = True 2024-01-27T03:41:43.144 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify missing on start = True 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: osd fast shutdown = False 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock override recovery settings = True 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue = debug_random 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue cut off = debug_random 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: osd shutdown pgref assert = True 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-01-27T03:41:43.145 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2024-01-27T03:41:43.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: mon min osdmap epochs = 50 2024-01-27T03:41:43.147 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd crush smoke test = False 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune interval = 2 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune min = 15 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune txsize = 2 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub interval = 300 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on pool no app = False 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: paxos service trim min = 10 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-01-27T03:41:43.148 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi110.stdout: debug objecter = 20 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi110.stdout: debug rados = 20 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi162.stdout:[global] 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: chdir = "" 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: auth supported = cephx 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.149 INFO:teuthology.orchestra.run.smithi162.stdout: filestore xattr use omap = true 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: mon clock drift allowed = 1.000 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: osd crush chooseleaf type = 0 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: auth debug = true 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: ms die on old message = true 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: ms die on bug = true 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: mon pg warn max object skew = 0 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: # disable pg_autoscaler by default for new pools 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.150 INFO:teuthology.orchestra.run.smithi162.stdout: osd pool default size = 2 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd allow primary affinity = true 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd allow pg remap = true 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on legacy crush tunables = false 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on crush straw calc version zero = false 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on no sortbitwise = false 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on osd down out interval zero = false 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on too few osds = false 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: mon_allow_pool_size_one = true 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-27T03:41:43.151 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: osd default data pool replay window = 5 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: mon allow pool delete = true 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: mon cluster log file level = debug 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: debug asserts on shutdown = true 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: mon health detail to clog = false 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: mon host = "172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.162" 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: mon client directed command retry = 5 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: mon election default strategy = 1 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: ms inject socket failures = 5000 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: ms type = async 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: osd heartbeat grace = 80 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: fsid = 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:41:43.152 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout:[osd] 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd journal size = 100 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd scrub load threshold = 5.0 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd scrub max interval = 600 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd mclock profile = high_recovery_ops 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd recover clone overlap = true 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd recovery max chunk = 1048576 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug shutdown = true 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug op order = true 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify stray on activate = true 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug trim objects = true 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd open classes on start = true 2024-01-27T03:41:43.153 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug pg log writeout = true 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: osd deep scrub update digest min age = 30 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: osd map max advance = 10 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: journal zero on create = true 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: filestore ondisk finisher threads = 3 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: filestore apply finisher threads = 3 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: bdev debug aio = true 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug misdirected ops = true 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: bdev async discard = True 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: bdev enable discard = True 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore allocator = bitmap 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore block size = 96636764160 2024-01-27T03:41:43.154 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore fsck on mount = True 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore zero block detection = True 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: debug bluefs = 1 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: debug bluestore = 1 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: debug monc = 20 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: debug osd = 20 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: debug rocksdb = 10 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd backfillfull_ratio = 0.85 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd full ratio = 0.9 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd nearfull ratio = 0.8 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: osd class default list = * 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: osd class load list = * 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: osd client watch timeout = 120 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify cached snaps = True 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify missing on start = True 2024-01-27T03:41:43.155 INFO:teuthology.orchestra.run.smithi162.stdout: osd failsafe full ratio = 0.95 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: osd fast shutdown = False 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: osd mclock override recovery settings = True 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: osd objectstore = bluestore 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: osd op queue = debug_random 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: osd op queue cut off = debug_random 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: osd shutdown pgref assert = True 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr] 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: debug mgr = 20 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: debug mon = 20 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: debug auth = 20 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min pgs per osd = 4 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min bytes per osd = 10 2024-01-27T03:41:43.156 INFO:teuthology.orchestra.run.smithi162.stdout: mgr/telemetry/nag = false 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout:[mon] 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: debug mon = 20 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: debug paxos = 20 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: debug auth = 20 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: mon data avail warn = 5 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: mon mgr mkfs grace = 240 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min pgs per osd = 4 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd reporter subtree level = osd 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd prime pg temp = true 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min bytes per osd = 10 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-27T03:41:43.157 INFO:teuthology.orchestra.run.smithi162.stdout: auth mon ticket ttl = 660 # 11m 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: auth service ticket ttl = 240 # 4m 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: # don't complain about insecure global_id in the test suite 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: # 1m isn't quite enough 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon_down_mkfs_grace = 2m 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_filestore_osds = false 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon min osdmap epochs = 50 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd crush smoke test = False 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon osdmap full prune interval = 2 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon osdmap full prune min = 15 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon osdmap full prune txsize = 2 2024-01-27T03:41:43.158 INFO:teuthology.orchestra.run.smithi162.stdout: mon scrub interval = 300 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on pool no app = False 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: paxos service trim min = 10 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout:[client] 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: rgw cache enabled = true 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: rgw enable ops log = true 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: rgw enable usage log = true 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: debug monc = 20 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: debug objecter = 20 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout: debug rados = 20 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.a] 2024-01-27T03:41:43.159 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.c] 2024-01-27T03:41:43.160 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.b] 2024-01-27T03:41:43.161 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-27T03:41:43.161 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-27T03:41:43.255 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-27T03:41:43.256 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:43.256 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-27T03:41:43.270 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:43.270 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-27T03:41:43.325 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T03:41:43.325 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:43.325 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-27T03:41:43.325 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-27T03:41:43.404 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:41:43.404 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:41:43.458 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T03:41:43.458 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:41:43.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-27T03:41:43.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-27T03:41:43.499 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:41:43.499 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:41:43.554 INFO:tasks.ceph:Setting up mon nodes... 2024-01-27T03:41:43.555 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-27T03:41:43.555 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-27T03:41:43.609 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-27T03:41:43.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-27T03:41:43.680 INFO:teuthology.orchestra.run.smithi162.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-27T03:41:43.683 INFO:tasks.ceph:Setting up mds nodes... 2024-01-27T03:41:43.683 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-27T03:41:43.683 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-27T03:41:43.726 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-27T03:41:43.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-27T03:41:43.784 INFO:teuthology.orchestra.run.smithi162.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-27T03:41:43.799 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-27T03:41:43.799 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi162.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-27T03:41:43.799 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-27T03:41:43.825 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:43.826 INFO:tasks.ceph:role: osd.0 2024-01-27T03:41:43.826 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T03:41:43.826 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-27T03:41:43.892 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-27T03:41:43.892 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-27T03:41:43.892 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-27T03:41:43.892 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-27T03:41:43.892 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-27T03:41:43.892 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-01-27T03:41:43.892 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-27T03:41:43.893 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-27T03:41:43.893 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-27T03:41:43.893 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-27T03:41:43.925 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-01-27T03:41:43.926 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-01-27T03:41:43.927 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-27T03:41:43.997 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-27T03:41:44.067 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-27T03:41:44.135 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:44.135 INFO:tasks.ceph:role: osd.1 2024-01-27T03:41:44.135 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T03:41:44.135 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-27T03:41:44.201 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-27T03:41:44.201 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-27T03:41:44.201 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-27T03:41:44.201 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-27T03:41:44.201 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-27T03:41:44.201 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-01-27T03:41:44.201 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-27T03:41:44.202 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-27T03:41:44.202 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-27T03:41:44.202 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-27T03:41:44.233 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-01-27T03:41:44.235 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-01-27T03:41:44.235 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-27T03:41:44.316 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-27T03:41:44.388 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-27T03:41:44.455 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:44.456 INFO:tasks.ceph:role: osd.2 2024-01-27T03:41:44.456 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T03:41:44.456 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-27T03:41:44.522 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-27T03:41:44.522 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-27T03:41:44.522 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-27T03:41:44.523 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-27T03:41:44.523 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-27T03:41:44.523 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-01-27T03:41:44.523 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-27T03:41:44.523 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-27T03:41:44.523 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-27T03:41:44.523 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-27T03:41:44.558 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-01-27T03:41:44.559 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-01-27T03:41:44.560 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-27T03:41:44.631 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-27T03:41:44.701 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-27T03:41:44.769 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:44.769 INFO:tasks.ceph:role: osd.3 2024-01-27T03:41:44.769 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T03:41:44.769 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-27T03:41:44.834 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-27T03:41:44.835 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-27T03:41:44.835 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-27T03:41:44.835 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-27T03:41:44.835 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-27T03:41:44.835 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-01-27T03:41:44.835 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-27T03:41:44.835 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-27T03:41:44.835 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-27T03:41:44.835 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-27T03:41:44.867 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-01-27T03:41:44.869 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-01-27T03:41:44.869 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-27T03:41:44.939 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-27T03:41:45.010 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:41:45.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:45.108+0000 7fe9a0a99640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-27T03:41:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:45.108+0000 7fe9a0a99640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-27T03:41:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:45.108+0000 7fe9a0a99640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-27T03:41:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:45.108+0000 7fe9a0a99640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-27T03:41:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:45.108+0000 7fe9a0a99640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-27T03:41:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:45.108+0000 7fe9a0a99640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-27T03:41:47.949 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-27T03:41:47.981 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:41:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:48.071+0000 7f71f58f7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-27T03:41:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:48.071+0000 7f71f58f7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-27T03:41:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:48.071+0000 7f71f58f7640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-27T03:41:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:48.071+0000 7f71f58f7640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-27T03:41:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:48.071+0000 7f71f58f7640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-27T03:41:48.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:48.071+0000 7f71f58f7640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-27T03:41:50.921 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-27T03:41:50.952 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:41:51.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:51.045+0000 7f4e0bbf9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-27T03:41:51.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:51.045+0000 7f4e0bbf9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-27T03:41:51.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:51.045+0000 7f4e0bbf9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-27T03:41:51.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:51.045+0000 7f4e0bbf9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-27T03:41:51.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:51.045+0000 7f4e0bbf9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-27T03:41:51.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:51.045+0000 7f4e0bbf9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-27T03:41:53.879 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-27T03:41:53.910 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:41:54.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:54.000+0000 7f9f85373640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-27T03:41:54.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:54.000+0000 7f9f85373640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-27T03:41:54.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:54.000+0000 7f9f85373640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-27T03:41:54.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:54.000+0000 7f9f85373640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-27T03:41:54.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:54.000+0000 7f9f85373640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-27T03:41:54.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:41:54.000+0000 7f9f85373640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-27T03:41:56.838 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-27T03:41:56.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-27T03:41:56.900 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:56.900 INFO:tasks.ceph:role: osd.4 2024-01-27T03:41:56.900 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T03:41:56.900 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-27T03:41:56.979 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-27T03:41:57.495 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-01-27T03:41:57.497 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-01-27T03:41:57.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-27T03:41:57.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-27T03:41:57.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-27T03:41:57.727 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:57.727 INFO:tasks.ceph:role: osd.5 2024-01-27T03:41:57.727 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T03:41:57.727 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-27T03:41:57.802 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-27T03:41:57.802 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-27T03:41:57.802 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-27T03:41:57.802 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-27T03:41:57.802 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-27T03:41:57.802 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-01-27T03:41:57.802 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-27T03:41:57.802 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-27T03:41:57.803 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-27T03:41:57.803 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-27T03:41:58.259 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-01-27T03:41:58.261 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-01-27T03:41:58.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-27T03:41:58.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-27T03:41:58.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-27T03:41:58.457 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:58.457 INFO:tasks.ceph:role: osd.6 2024-01-27T03:41:58.457 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T03:41:58.457 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-27T03:41:58.533 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-27T03:41:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-27T03:41:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-27T03:41:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-27T03:41:58.533 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-27T03:41:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-01-27T03:41:58.533 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-27T03:41:58.534 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-27T03:41:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-27T03:41:58.534 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-27T03:41:59.028 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-01-27T03:41:59.030 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-01-27T03:41:59.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-27T03:41:59.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-27T03:41:59.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-27T03:41:59.254 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-27T03:41:59.254 INFO:tasks.ceph:role: osd.7 2024-01-27T03:41:59.254 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T03:41:59.255 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-27T03:41:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-27T03:41:59.331 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-27T03:41:59.331 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-27T03:41:59.331 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-27T03:41:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-27T03:41:59.331 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-01-27T03:41:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-27T03:41:59.332 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-27T03:41:59.332 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-27T03:41:59.332 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-27T03:41:59.836 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-01-27T03:41:59.838 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-01-27T03:41:59.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-27T03:41:59.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-27T03:42:00.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:42:00.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:00.140+0000 7f7f5e215640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-27T03:42:00.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:00.140+0000 7f7f5e215640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-27T03:42:00.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:00.140+0000 7f7f5e215640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-27T03:42:00.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:00.140+0000 7f7f5e215640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-27T03:42:00.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:00.140+0000 7f7f5e215640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-27T03:42:00.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:00.141+0000 7f7f5e215640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-27T03:42:02.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-27T03:42:03.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:42:03.118 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:03.117+0000 7f894c6f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-27T03:42:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:03.117+0000 7f894c6f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-27T03:42:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:03.117+0000 7f894c6f9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-27T03:42:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:03.117+0000 7f894c6f9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-27T03:42:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:03.117+0000 7f894c6f9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-27T03:42:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:03.118+0000 7f894c6f9640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-27T03:42:05.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-27T03:42:05.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:42:06.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:06.076+0000 7f64cae99640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-27T03:42:06.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:06.076+0000 7f64cae99640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-27T03:42:06.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:06.076+0000 7f64cae99640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-27T03:42:06.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:06.076+0000 7f64cae99640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-27T03:42:06.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:06.076+0000 7f64cae99640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-27T03:42:06.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:06.077+0000 7f64cae99640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-27T03:42:08.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-27T03:42:08.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:42:09.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:09.042+0000 7fba61ef9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-27T03:42:09.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:09.043+0000 7fba61ef9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-27T03:42:09.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:09.043+0000 7fba61ef9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-27T03:42:09.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:09.043+0000 7fba61ef9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-27T03:42:09.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:09.043+0000 7fba61ef9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-27T03:42:09.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:09.043+0000 7fba61ef9640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-27T03:42:11.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-27T03:42:11.915 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-27T03:42:11.915 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:11.916 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-27T03:42:11.947 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:11.947 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-27T03:42:12.017 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:12.017 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-27T03:42:12.086 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:12.086 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-27T03:42:12.154 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:12.154 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-27T03:42:12.222 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:12.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-27T03:42:12.253 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:12.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-27T03:42:12.323 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:12.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-27T03:42:12.394 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:12.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-27T03:42:12.463 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:12.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-27T03:42:12.531 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:12.532 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-27T03:42:12.548 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:12.548 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-27T03:42:12.587 INFO:tasks.ceph:Adding keys to all mons... 2024-01-27T03:42:12.588 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-27T03:42:12.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-27T03:42:12.631 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2024-01-27T03:42:12.631 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQB3e7RlvoxaJBAA1kGj8hzjN6gfAPDfsR9AVQ== 2024-01-27T03:42:12.631 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2024-01-27T03:42:12.631 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQB5e7RlvKaEBhAAmFQ2zy2HLX7LXVE9MtvHmg== 2024-01-27T03:42:12.631 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2024-01-27T03:42:12.631 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQB8e7RlegxTBBAA4LVdUh32tdTcbS4xZZ9ytA== 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQB/e7Rl/dTBAhAATSr9JpHEk2YryNMwQiluog== 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCCe7RlhU0XABAANmac7AYxZpRH4RvTil+meQ== 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQB3e7RlIQ2XKBAA8KTnHVFMeErzOFGuAf4TLw== 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCIe7RlXXVzCBAAnvro3MhFu5W/MelhpGo0Vg== 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCLe7RlBgQTBxAAnAj9SIUwQhZlWuB7dUJ4gg== 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCOe7RlcPmjBBAANveGDnx57Qahagubi1kJCg== 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCRe7Rl+1eiAhAAQo8Ct73CwBSOwvj5rZRrJg== 2024-01-27T03:42:12.632 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2024-01-27T03:42:12.633 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQB3e7RlYdlMKxAAhW3dkZBsGFWhBz7AmezVIg== 2024-01-27T03:42:12.633 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2024-01-27T03:42:12.633 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQB3e7Rl3Lm/LhAAWRUm65lWR5HJH75O2edTtA== 2024-01-27T03:42:12.654 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr.y] 2024-01-27T03:42:12.654 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQB3e7RlvoxaJBAA1kGj8hzjN6gfAPDfsR9AVQ== 2024-01-27T03:42:12.654 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.0] 2024-01-27T03:42:12.654 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQB5e7RlvKaEBhAAmFQ2zy2HLX7LXVE9MtvHmg== 2024-01-27T03:42:12.654 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.1] 2024-01-27T03:42:12.654 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQB8e7RlegxTBBAA4LVdUh32tdTcbS4xZZ9ytA== 2024-01-27T03:42:12.654 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.2] 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQB/e7Rl/dTBAhAATSr9JpHEk2YryNMwQiluog== 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.3] 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQCCe7RlhU0XABAANmac7AYxZpRH4RvTil+meQ== 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr.x] 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQB3e7RlIQ2XKBAA8KTnHVFMeErzOFGuAf4TLw== 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.4] 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQCIe7RlXXVzCBAAnvro3MhFu5W/MelhpGo0Vg== 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.5] 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQCLe7RlBgQTBxAAnAj9SIUwQhZlWuB7dUJ4gg== 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.6] 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQCOe7RlcPmjBBAANveGDnx57Qahagubi1kJCg== 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.7] 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQCRe7Rl+1eiAhAAQo8Ct73CwBSOwvj5rZRrJg== 2024-01-27T03:42:12.655 INFO:teuthology.orchestra.run.smithi162.stdout:[client.0] 2024-01-27T03:42:12.656 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQB3e7RlYdlMKxAAhW3dkZBsGFWhBz7AmezVIg== 2024-01-27T03:42:12.656 INFO:teuthology.orchestra.run.smithi162.stdout:[client.1] 2024-01-27T03:42:12.656 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQB3e7Rl3Lm/LhAAWRUm65lWR5HJH75O2edTtA== 2024-01-27T03:42:12.657 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-27T03:42:12.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-27T03:42:12.751 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:12.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:12.805 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:12.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:12.859 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:12.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:12.913 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:12.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:12.967 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-27T03:42:12.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-27T03:42:13.020 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:13.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:13.075 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:13.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:13.128 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:13.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:13.182 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:13.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-27T03:42:13.235 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-27T03:42:13.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-27T03:42:13.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-27T03:42:13.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-27T03:42:13.350 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-27T03:42:13.350 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-27T03:42:13.376 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-27T03:42:13.473 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-27T03:42:13.499 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-27T03:42:13.566 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-27T03:42:13.662 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-27T03:42:13.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-27T03:42:13.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-27T03:42:13.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-27T03:42:13.839 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:42:13.841 DEBUG:teuthology.orchestra.run.smithi162:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-27T03:42:13.893 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-27T03:42:13.894 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-27T03:42:13.894 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-27T03:42:13.894 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-01-27T03:42:13.896 INFO:tasks.ceph.mon.a:Started 2024-01-27T03:42:13.896 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-27T03:42:13.897 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-27T03:42:13.897 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-01-27T03:42:13.898 INFO:tasks.ceph.mon.c:Started 2024-01-27T03:42:13.899 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-27T03:42:13.899 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-27T03:42:13.899 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-01-27T03:42:13.936 INFO:tasks.ceph.mon.b:Started 2024-01-27T03:42:13.937 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-27T03:42:13.937 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-27T03:42:13.937 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-01-27T03:42:13.939 INFO:tasks.ceph.mgr.y:Started 2024-01-27T03:42:13.939 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-27T03:42:13.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-01-27T03:42:13.941 INFO:tasks.ceph.mgr.x:Started 2024-01-27T03:42:13.941 INFO:tasks.ceph:Setting crush tunables to default 2024-01-27T03:42:13.942 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd crush tunables default 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.041+0000 7fe03b3d5640 10 monclient: get_monmap_and_config 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.041+0000 7fe03b3d5640 10 monclient: build_initial_monmap 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.041+0000 7fe03b3d5640 10 monclient: monmap: 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:42:14.042710+0000 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:42:14.042710+0000 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:14.043 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:14.044 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:14.044 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:14.044 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:14.044 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:14.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 1 Processor -- start 2024-01-27T03:42:14.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 1 -- start start 2024-01-27T03:42:14.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 10 monclient: init 2024-01-27T03:42:14.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:14.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-01-27T03:42:14.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0340b4680 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:14.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 0x7fe0340b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 10 monclient(hunting): picked mon.noname-b con 0x7fe0340b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 0x7fe0340b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 10 monclient(hunting): picked mon.noname-c con 0x7fe0340b3b30 addr v2:172.21.15.162:3300/0 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.042+0000 7fe03b3d5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340afbe0 con 0x7fe0340b4680 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe03b3d5640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe03b3d5640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe034142240 con 0x7fe0340b3b30 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe03b3d5640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe03b3d5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0340b0890 con 0x7fe0340b4a60 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe03b3d5640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe03b3d5640 10 monclient(hunting): authenticate will time out at 1417.396978s 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe038970640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 msgr2=0x7fe0340b1470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe038970640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 0x7fe0340b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe039972640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 msgr2=0x7fe0340b19b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.162:3300/0 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe039972640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 0x7fe0340b19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T03:42:14.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.043+0000 7fe039171640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe0340b4680 legacy=0x7fe034142970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-27T03:42:14.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.244+0000 7fe038970640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 msgr2=0x7fe0340b1470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-27T03:42:14.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.244+0000 7fe039171640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe0340b4680 legacy=0x7fe034142970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-27T03:42:14.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.244+0000 7fe038970640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 0x7fe0340b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T03:42:14.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.244+0000 7fe039972640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 msgr2=0x7fe0340b19b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.162:3300/0 2024-01-27T03:42:14.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.244+0000 7fe039972640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 0x7fe0340b19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T03:42:14.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.645+0000 7fe038970640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 msgr2=0x7fe0340b1470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-27T03:42:14.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.645+0000 7fe039171640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe0340b4680 legacy=0x7fe034142970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-27T03:42:14.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.645+0000 7fe038970640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 0x7fe0340b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T03:42:14.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.645+0000 7fe039972640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 msgr2=0x7fe0340b19b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.162:3300/0 2024-01-27T03:42:14.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:14.645+0000 7fe039972640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 0x7fe0340b19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T03:42:15.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:15.446+0000 7fe038970640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 msgr2=0x7fe0340b1470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-01-27T03:42:15.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:15.446+0000 7fe039171640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe0340b4680 legacy=0x7fe034142970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-01-27T03:42:15.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:15.446+0000 7fe038970640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 0x7fe0340b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T03:42:15.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:15.446+0000 7fe039972640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 msgr2=0x7fe0340b19b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.162:3300/0 2024-01-27T03:42:15.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:15.446+0000 7fe039972640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 0x7fe0340b19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T03:42:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 10 monclient(hunting): tick 2024-01-27T03:42:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 10 monclient(hunting): _check_auth_tickets 2024-01-27T03:42:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 1 monclient(hunting): continuing hunt 2024-01-27T03:42:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 10 monclient(hunting): _reopen_session rank -1 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 msgr2=0x7fe0340b1470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0340b4a60 0x7fe0340b1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 msgr2=0x7fe0340b19b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0340b3b30 0x7fe0340b19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe0340b4680 legacy=0x7fe034142970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 10 monclient: _add_conns ranks=[3,1] 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 0x7fe018005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-a con 0x7fe0180036c0 addr v2:172.21.15.110:3300/0 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe018006190 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340b0b00 con 0x7fe018006190 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0340b0890 con 0x7fe0180036c0 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe02affd640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe039171640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-27T03:42:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe039171640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T03:42:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.043+0000 7fe038970640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe018006190 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-01-27T03:42:17.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.244+0000 7fe039171640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-27T03:42:17.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.244+0000 7fe039171640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T03:42:17.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.244+0000 7fe038970640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe018006190 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-01-27T03:42:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.645+0000 7fe039171640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-27T03:42:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.645+0000 7fe039171640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T03:42:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:17.645+0000 7fe038970640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe018006190 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-01-27T03:42:18.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:18.446+0000 7fe039171640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-27T03:42:18.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:18.446+0000 7fe039171640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T03:42:18.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:18.446+0000 7fe038970640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe018006190 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-01-27T03:42:20.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient(hunting): tick 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient(hunting): _check_auth_tickets 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 1 monclient(hunting): continuing hunt 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient(hunting): _reopen_session rank -1 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0180036c0 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe018006190 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe039972640 1 -- reap_dead start 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 0x7fe018005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-a con 0x7fe018009f00 addr v2:172.21.15.110:3300/0 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe01800a3b0 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:20.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 0x7fe01800d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-c con 0x7fe01800a8a0 addr v2:172.21.15.162:3300/0 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340afbe0 con 0x7fe01800a3b0 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0340b0890 con 0x7fe018009f00 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01800e660 con 0x7fe01800a8a0 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe02affd640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe039171640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe039171640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe039972640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 msgr2=0x7fe01800d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.162:3300/0 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe038970640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe01800a3b0 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-01-27T03:42:20.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.043+0000 7fe039972640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 0x7fe01800d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T03:42:20.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.244+0000 7fe039171640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-27T03:42:20.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.244+0000 7fe039171640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T03:42:20.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.244+0000 7fe038970640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe01800a3b0 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-01-27T03:42:20.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.244+0000 7fe039972640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 msgr2=0x7fe01800d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.162:3300/0 2024-01-27T03:42:20.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.244+0000 7fe039972640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 0x7fe01800d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T03:42:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.645+0000 7fe039171640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-27T03:42:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.645+0000 7fe038970640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe01800a3b0 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-01-27T03:42:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.645+0000 7fe039171640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T03:42:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.645+0000 7fe039972640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 msgr2=0x7fe01800d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.162:3300/0 2024-01-27T03:42:20.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:20.645+0000 7fe039972640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 0x7fe01800d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T03:42:20.869 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:00:06.814 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:42:21.032 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:00:06.988 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:42:21.032 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:00:06.989 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:42:21.175 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:00:07.120 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:42:21.352 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:00:07.308 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:42:21.353 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:00:07.309 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:42:21.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:21.446+0000 7fe039171640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-01-27T03:42:21.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:21.446+0000 7fe039171640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T03:42:21.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:21.446+0000 7fe039972640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 msgr2=0x7fe01800d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.162:3300/0 2024-01-27T03:42:21.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:21.446+0000 7fe039972640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 0x7fe01800d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T03:42:21.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:21.446+0000 7fe038970640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe01800a3b0 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-01-27T03:42:23.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 10 monclient(hunting): tick 2024-01-27T03:42:23.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 10 monclient(hunting): _check_auth_tickets 2024-01-27T03:42:23.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 1 monclient(hunting): continuing hunt 2024-01-27T03:42:23.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 10 monclient(hunting): _reopen_session rank -1 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 1 -- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 msgr2=0x7fe01800d0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe01800a8a0 0x7fe01800d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 msgr2=0x7fe018005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe018009f00 0x7fe018005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7fe01800a3b0 legacy=0x7fe018008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 10 monclient: _add_conns ranks=[2,0] 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-b con 0x7fe01800d610 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe01800e060 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340b0b00 con 0x7fe01800e060 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0340b0890 con 0x7fe01800d610 2024-01-27T03:42:23.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.043+0000 7fe02affd640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:23.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.053+0000 7fe039171640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:23.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.057+0000 7fe038970640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe01800e060 0x7fe0180128b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59956/0 (socket says 172.21.15.110:59956) 2024-01-27T03:42:23.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.057+0000 7fe038970640 1 -- 172.21.15.110:0/3735933414 learned_addr learned my addr 172.21.15.110:0/3735933414 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:42:23.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.059+0000 7fe039171640 10 monclient(hunting): get_auth_request con 0x7fe01800d610 auth_method 0 2024-01-27T03:42:23.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.059+0000 7fe039171640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:23.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.059+0000 7fe039171640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:23.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.059+0000 7fe039171640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:23.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.065+0000 7fe039171640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-27T03:42:23.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.065+0000 7fe039171640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-27T03:42:23.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.065+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-27T03:42:23.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.065+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T03:42:23.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.414+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:23.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.415+0000 7fe039171640 10 monclient(hunting): get_auth_request con 0x7fe01800d610 auth_method 0 2024-01-27T03:42:23.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.415+0000 7fe039171640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:23.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.415+0000 7fe039171640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:23.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.415+0000 7fe039171640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-27T03:42:23.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.427+0000 7fe039171640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-27T03:42:23.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.427+0000 7fe039171640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-27T03:42:23.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.427+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-27T03:42:23.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.427+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-27T03:42:23.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.832+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:23.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.847+0000 7fe039171640 10 monclient(hunting): get_auth_request con 0x7fe01800d610 auth_method 0 2024-01-27T03:42:23.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.847+0000 7fe039171640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:23.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.847+0000 7fe039171640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:23.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.847+0000 7fe039171640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-27T03:42:23.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.871+0000 7fe039171640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-27T03:42:23.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.871+0000 7fe039171640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-27T03:42:23.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.871+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-27T03:42:23.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:23.872+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-27T03:42:24.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:24.673+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:24.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:24.673+0000 7fe039171640 10 monclient(hunting): get_auth_request con 0x7fe01800d610 auth_method 0 2024-01-27T03:42:24.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:24.673+0000 7fe039171640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:24.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:24.673+0000 7fe039171640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:24.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:24.673+0000 7fe039171640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-27T03:42:24.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:24.674+0000 7fe039171640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-27T03:42:24.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:24.674+0000 7fe039171640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-27T03:42:24.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:24.674+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-27T03:42:24.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:24.674+0000 7fe039171640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-27T03:42:25.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:25.986+0000 7fe02bfff640 1 -- 172.21.15.110:0/3735933414 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781284989 0 0) 0x7fe0340b0b00 con 0x7fe01800e060 2024-01-27T03:42:25.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:25.986+0000 7fe02bfff640 10 monclient(hunting): _negotiate 2024-01-27T03:42:25.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:25.986+0000 7fe02bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:25.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:25.986+0000 7fe02bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:25.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:25.986+0000 7fe02bfff640 10 monclient(hunting): my global_id is 4098 2024-01-27T03:42:25.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:25.986+0000 7fe02bfff640 1 -- 172.21.15.110:0/3735933414 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe020003520 con 0x7fe01800e060 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient(hunting): tick 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient(hunting): _check_auth_tickets 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 1 monclient(hunting): continuing hunt 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient(hunting): _reopen_session rank -1 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe01800d610 0x7fe01800d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 1 -- 172.21.15.110:0/3735933414 >> v1:172.21.15.110:6789/0 conn(0x7fe01800e060 legacy=0x7fe0180128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe039972640 1 -- 172.21.15.110:0/3735933414 reap_dead start 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe018009f00 0x7fe01800d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:26.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-b con 0x7fe018009f00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 1 --2- 172.21.15.110:0/3735933414 >> v2:172.21.15.110:3300/0 conn(0x7fe0180140f0 0x7fe018005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-a con 0x7fe0180140f0 addr v2:172.21.15.110:3300/0 2024-01-27T03:42:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe018014620 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 1 -- 172.21.15.110:0/3735933414 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340afbe0 con 0x7fe018014620 2024-01-27T03:42:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 1 -- 172.21.15.110:0/3735933414 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0340b0890 con 0x7fe0180140f0 2024-01-27T03:42:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 1 -- 172.21.15.110:0/3735933414 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01800e660 con 0x7fe018009f00 2024-01-27T03:42:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.043+0000 7fe02affd640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:26.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.060+0000 7fe038970640 1 --2- 172.21.15.110:0/3735933414 >> v2:172.21.15.110:3300/0 conn(0x7fe0180140f0 0x7fe018005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:26.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.069+0000 7fe038970640 10 monclient(hunting): get_auth_request con 0x7fe0180140f0 auth_method 0 2024-01-27T03:42:26.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.069+0000 7fe038970640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:26.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.069+0000 7fe038970640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:26.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.069+0000 7fe038970640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:26.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.075+0000 7fe038970640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:26.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.075+0000 7fe038970640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:26.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.075+0000 7fe038970640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.077+0000 7fe038970640 10 monclient(hunting): handle_auth_done global_id 4107 payload 274 2024-01-27T03:42:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.077+0000 7fe038970640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe018009f00 msgr2=0x7fe01800d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.077+0000 7fe038970640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe018009f00 0x7fe01800d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.077+0000 7fe038970640 1 -- 172.21.15.110:0/3735933414 >> v1:172.21.15.110:6789/0 conn(0x7fe018014620 legacy=0x7fe0180107b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.077+0000 7fe038970640 10 monclient: _finish_hunting 0 2024-01-27T03:42:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.077+0000 7fe038970640 1 monclient: found mon.noname-a 2024-01-27T03:42:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.077+0000 7fe038970640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.077+0000 7fe038970640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:26.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.077+0000 7fe038970640 1 -- 172.21.15.110:0/3735933414 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe018014be0 con 0x7fe0180140f0 2024-01-27T03:42:26.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.095+0000 7fe038970640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0180140f0 0x7fe018005b40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe024003bf0 tx=0x7fe02402c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab5395cd4be3d77 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:26.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 1 -- 172.21.15.110:0/3735933414 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024035470 con 0x7fe0180140f0 2024-01-27T03:42:26.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:26.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:26.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 10 monclient: dump: 2024-01-27T03:42:26.140 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:42:26.140 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:26.140 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:26.140 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 10 monclient: _finish_auth 0 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 10 monclient: _check_auth_tickets 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 1 -- 172.21.15.110:0/3735933414 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0240355d0 con 0x7fe0180140f0 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 1 -- 172.21.15.110:0/3735933414 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024035980 con 0x7fe0180140f0 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe03b3d5640 5 monclient: authenticate success, global_id 4107 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe03b3d5640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:26.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe03b3d5640 4 set_mon_vals no callback set 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.138+0000 7fe02bfff640 10 monclient: dump: 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:26.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 10 monclient: shutdown 2024-01-27T03:42:26.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:26.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0180140f0 msgr2=0x7fe018005b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:26.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0180140f0 0x7fe018005b40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe024003bf0 tx=0x7fe02402c110 comp rx=0 tx=0).stop 2024-01-27T03:42:26.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3735933414 shutdown_connections 2024-01-27T03:42:26.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0180140f0 0x7fe018005b40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:26.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 1 --2- 172.21.15.110:0/3735933414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe018009f00 0x7fe01800d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:26.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3735933414 >> 172.21.15.110:0/3735933414 conn(0x7fe034066490 msgr2=0x7fe0340668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:26.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3735933414 shutdown_connections 2024-01-27T03:42:26.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.139+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3735933414 wait complete. 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 10 monclient: build_initial_monmap 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 10 monclient: monmap: 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:26.145 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 1 librados: starting msgr at 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 1 librados: starting objecter 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 1 Processor -- start 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 1 -- start start 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 1 librados: setting wanted keys 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 1 librados: calling monclient init 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 10 monclient: init 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe018009f00 0x7fe03413c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.140+0000 7fe03b3d5640 10 monclient(hunting): picked mon.noname-a con 0x7fe018009f00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0180140f0 0x7fe03413cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 10 monclient(hunting): picked mon.noname-b con 0x7fe0180140f0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe018014620 0x7fe0341d6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:26.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 10 monclient(hunting): picked mon.noname-c con 0x7fe018014620 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:26.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:26.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03413d240 con 0x7fe018009f00 2024-01-27T03:42:26.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:26.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03411fc90 con 0x7fe0180140f0 2024-01-27T03:42:26.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:26.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0340ac550 con 0x7fe018014620 2024-01-27T03:42:26.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:26.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.141+0000 7fe03b3d5640 10 monclient(hunting): authenticate will time out at 1429.495003s 2024-01-27T03:42:26.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.146+0000 7fe038970640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0180140f0 0x7fe03413cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:26.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.157+0000 7fe039171640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe018009f00 0x7fe03413c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:26.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.159+0000 7fe039972640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe018014620 0x7fe0341d6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:26.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.159+0000 7fe038970640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0180140f0 0x7fe03413cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:55114/0 (socket says 172.21.15.110:55114) 2024-01-27T03:42:26.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.159+0000 7fe038970640 1 -- 172.21.15.110:0/3689042961 learned_addr learned my addr 172.21.15.110:0/3689042961 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:42:26.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.159+0000 7fe038970640 10 monclient(hunting): get_auth_request con 0x7fe0180140f0 auth_method 0 2024-01-27T03:42:26.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.159+0000 7fe038970640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:26.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.159+0000 7fe038970640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:26.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.159+0000 7fe038970640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:26.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.163+0000 7fe038970640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:26.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.163+0000 7fe038970640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:26.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.163+0000 7fe038970640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:26.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 10 monclient(hunting): handle_auth_done global_id 4105 payload 824 2024-01-27T03:42:26.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 1 -- 172.21.15.110:0/3689042961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe018014620 msgr2=0x7fe0341d6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:26.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 1 --2- 172.21.15.110:0/3689042961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe018014620 0x7fe0341d6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:26.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 1 -- 172.21.15.110:0/3689042961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe018009f00 msgr2=0x7fe03413c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:26.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 1 --2- 172.21.15.110:0/3689042961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe018009f00 0x7fe03413c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:26.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 10 monclient: _finish_hunting 0 2024-01-27T03:42:26.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 1 monclient: found mon.noname-b 2024-01-27T03:42:26.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:26.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:42:26.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.166+0000 7fe038970640 1 -- 172.21.15.110:0/3689042961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe020003520 con 0x7fe0180140f0 2024-01-27T03:42:26.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.203+0000 7fe038970640 1 --2- 172.21.15.110:0/3689042961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0180140f0 0x7fe03413cd00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe02402c150 tx=0x7fe024003b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:26.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.203+0000 7fe029ffb640 10 client.?.objecter ms_handle_connect 0x7fe0180140f0 2024-01-27T03:42:26.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.203+0000 7fe029ffb640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 1 -- 172.21.15.110:0/3689042961 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024033780 con 0x7fe0180140f0 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 10 monclient: dump: 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:26.239 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 10 monclient: _finish_auth 0 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 10 monclient: _check_auth_tickets 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 1 -- 172.21.15.110:0/3689042961 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe024033900 con 0x7fe0180140f0 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 1 -- 172.21.15.110:0/3689042961 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024033cb0 con 0x7fe0180140f0 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe029ffb640 10 monclient: dump: 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe03b3d5640 5 monclient: authenticate success, global_id 4105 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe02b7fe640 4 set_mon_vals no callback set 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe03b3d5640 10 monclient: _renew_subs 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe03b3d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3689042961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0341d7510 con 0x7fe0180140f0 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe03b3d5640 10 client.4105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe03b3d5640 10 monclient: _renew_subs 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe03b3d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3689042961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0341d79f0 con 0x7fe0180140f0 2024-01-27T03:42:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.237+0000 7fe03b3d5640 1 librados: init done 2024-01-27T03:42:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.238+0000 7fe03b3d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.238+0000 7fe03b3d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.238+0000 7fe03b3d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.238+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3689042961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03411fad0 con 0x7fe0180140f0 2024-01-27T03:42:26.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.410+0000 7fe029ffb640 1 -- 172.21.15.110:0/3689042961 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe024032070 con 0x7fe0180140f0 2024-01-27T03:42:26.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.416+0000 7fe029ffb640 1 -- 172.21.15.110:0/3689042961 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe024040400 con 0x7fe0180140f0 2024-01-27T03:42:26.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.416+0000 7fe029ffb640 10 client.4105.objecter ms_dispatch 0x7fe034000b90 osd_map(1..1 src has 1..1) v4 2024-01-27T03:42:26.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.416+0000 7fe029ffb640 3 client.4105.objecter handle_osd_map got epochs [1,1] > 0 2024-01-27T03:42:26.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.416+0000 7fe029ffb640 3 client.4105.objecter handle_osd_map decoding full epoch 1 2024-01-27T03:42:26.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.416+0000 7fe029ffb640 20 client.4105.objecter dump_active .. 0 homeless 2024-01-27T03:42:26.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.416+0000 7fe029ffb640 1 -- 172.21.15.110:0/3689042961 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7fe024045070 con 0x7fe0180140f0 2024-01-27T03:42:26.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.416+0000 7fe029ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:26.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.416+0000 7fe029ffb640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:26.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.478+0000 7fe03b3d5640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-27T03:42:26.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.478+0000 7fe03b3d5640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-27T03:42:26.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.478+0000 7fe03b3d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:26.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.478+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3689042961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe03404ea50 con 0x7fe0180140f0 2024-01-27T03:42:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.711+0000 7fe029ffb640 1 -- 172.21.15.110:0/3689042961 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe024003960 con 0x7fe0180140f0 2024-01-27T03:42:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.711+0000 7fe029ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-27T03:42:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.711+0000 7fe029ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-01-27T03:42:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2024-01-27T03:42:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 10 librados: watch_flush enter 2024-01-27T03:42:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 10 librados: watch_flush exit 2024-01-27T03:42:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 20 client.4105.objecter shutdown clearing up homeless session... 2024-01-27T03:42:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 10 client.4105.objecter successfully canceled tick 2024-01-27T03:42:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 10 monclient: shutdown 2024-01-27T03:42:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3689042961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0180140f0 msgr2=0x7fe03413cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 --2- 172.21.15.110:0/3689042961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0180140f0 0x7fe03413cd00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe02402c150 tx=0x7fe024003b80 comp rx=0 tx=0).stop 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3689042961 shutdown_connections 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 --2- 172.21.15.110:0/3689042961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe018014620 0x7fe0341d6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 --2- 172.21.15.110:0/3689042961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0180140f0 0x7fe03413cd00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 --2- 172.21.15.110:0/3689042961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe018009f00 0x7fe03413c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3689042961 >> 172.21.15.110:0/3689042961 conn(0x7fe034066490 msgr2=0x7fe034066870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3689042961 shutdown_connections 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 -- 172.21.15.110:0/3689042961 wait complete. 2024-01-27T03:42:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:26.712+0000 7fe03b3d5640 1 librados: shutdown 2024-01-27T03:42:26.724 INFO:tasks.ceph:check_enable_crimson: False 2024-01-27T03:42:26.724 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-27T03:42:26.724 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:26.724 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-27T03:42:26.750 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:26.751 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-27T03:42:26.759 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:26.759+0000 7f8733841040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T03:42:26.813 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:26.813+0000 7f8733841040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T03:42:26.817 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:26.818 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-27T03:42:26.885 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:42:26.885 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-27T03:42:26.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:26.946+0000 7f1407c0f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-27T03:42:26.952 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:26.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-27T03:42:26.979 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:26.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-27T03:42:27.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:27.003+0000 7f1407c0f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-27T03:42:27.046 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:27.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-27T03:42:27.112 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-27T03:42:27.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-27T03:42:27.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 112d1665-ee1b-41ea-85fc-4165aa26818e 0 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.321+0000 7fe4b53e9640 10 monclient: get_monmap_and_config 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.321+0000 7fe4b53e9640 10 monclient: build_initial_monmap 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.321+0000 7fe4b53e9640 10 monclient: monmap: 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:42:27.322243+0000 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:42:27.322243+0000 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:27.322 INFO:teuthology.orchestra.run.smithi162.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:27.323 INFO:teuthology.orchestra.run.smithi162.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:27.323 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.322+0000 7fe4b53e9640 1 Processor -- start 2024-01-27T03:42:27.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.322+0000 7fe4b53e9640 1 -- start start 2024-01-27T03:42:27.324 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.322+0000 7fe4b53e9640 10 monclient: init 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.322+0000 7fe4b53e9640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.322+0000 7fe4b53e9640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.322+0000 7fe4b53e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4b00b4b00 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.322+0000 7fe4b53e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4b013f730 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.322+0000 7fe4b53e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0142770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.322+0000 7fe4b53e9640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b00b3db0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.323+0000 7fe4b53e9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b00aaf30 con 0x7fe4b013f730 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.323+0000 7fe4b53e9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b00ab210 con 0x7fe4b00b4b00 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.323+0000 7fe4b53e9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.323+0000 7fe4b53e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b00b44d0 con 0x7fe4b00b3db0 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.323+0000 7fe4b53e9640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.323+0000 7fe4b53e9640 10 monclient(hunting): authenticate will time out at 1439.666873s 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.323+0000 7fe4af7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.323+0000 7fe4ae7fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe4b013f730 0x7fe4b0142060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.162:46536/0 (socket says 172.21.15.162:46536) 2024-01-27T03:42:27.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.323+0000 7fe4ae7fc640 1 -- 172.21.15.162:0/376442750 learned_addr learned my addr 172.21.15.162:0/376442750 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-01-27T03:42:27.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.324+0000 7fe4af7fe640 10 monclient(hunting): get_auth_request con 0x7fe4b00b3db0 auth_method 0 2024-01-27T03:42:27.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.324+0000 7fe4af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:27.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.324+0000 7fe4af7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:27.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.324+0000 7fe4af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.324+0000 7fe4af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.324+0000 7fe4af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.324+0000 7fe4af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.326+0000 7fe4adffb640 1 -- 172.21.15.162:0/376442750 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846667780 0 0) 0x7fe4b00aaf30 con 0x7fe4b013f730 2024-01-27T03:42:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.326+0000 7fe4adffb640 10 monclient(hunting): _negotiate 2024-01-27T03:42:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.326+0000 7fe4adffb640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.326+0000 7fe4adffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.326+0000 7fe4adffb640 10 monclient(hunting): my global_id is 4113 2024-01-27T03:42:27.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.326+0000 7fe4adffb640 1 -- 172.21.15.162:0/376442750 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe498003520 con 0x7fe4b013f730 2024-01-27T03:42:27.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.326+0000 7fe4af7fe640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-01-27T03:42:27.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.326+0000 7fe4af7fe640 1 -- 172.21.15.162:0/376442750 >> v1:172.21.15.162:6789/0 conn(0x7fe4b00b4b00 legacy=0x7fe4b013f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:27.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.327+0000 7fe4af7fe640 1 -- 172.21.15.162:0/376442750 >> v1:172.21.15.110:6789/0 conn(0x7fe4b013f730 legacy=0x7fe4b0142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:27.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.327+0000 7fe4af7fe640 10 monclient: _finish_hunting 0 2024-01-27T03:42:27.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.327+0000 7fe4af7fe640 1 monclient: found mon.noname-b 2024-01-27T03:42:27.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.327+0000 7fe4af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:27.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.327+0000 7fe4af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:42:27.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.327+0000 7fe4af7fe640 1 -- 172.21.15.162:0/376442750 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b0142cb0 con 0x7fe4b00b3db0 2024-01-27T03:42:27.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.329+0000 7fe4af7fe640 1 --2- 172.21.15.162:0/376442750 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0142770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8009dc0 tx=0x7fe4a802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ccd1fb6aa86e3a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:27.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 1 -- 172.21.15.162:0/376442750 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a802cb20 con 0x7fe4b00b3db0 2024-01-27T03:42:27.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:27.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:27.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 10 monclient: dump: 2024-01-27T03:42:27.335 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:27.335 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.335 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 10 monclient: _finish_auth 0 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 10 monclient: _check_auth_tickets 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 1 -- 172.21.15.162:0/376442750 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a802cc80 con 0x7fe4b00b3db0 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 1 -- 172.21.15.162:0/376442750 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8034440 con 0x7fe4b00b3db0 2024-01-27T03:42:27.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 5 monclient: authenticate success, global_id 4109 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 4 set_mon_vals no callback set 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4adffb640 10 monclient: dump: 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:27.337 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 10 monclient: shutdown 2024-01-27T03:42:27.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:27.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 1 -- 172.21.15.162:0/376442750 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 msgr2=0x7fe4b0142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:27.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 1 --2- 172.21.15.162:0/376442750 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0142770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8009dc0 tx=0x7fe4a802bd70 comp rx=0 tx=0).stop 2024-01-27T03:42:27.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 1 -- 172.21.15.162:0/376442750 shutdown_connections 2024-01-27T03:42:27.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 1 --2- 172.21.15.162:0/376442750 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0142770 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 1 -- 172.21.15.162:0/376442750 >> 172.21.15.162:0/376442750 conn(0x7fe4b0066490 msgr2=0x7fe4b00668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:27.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 1 -- 172.21.15.162:0/376442750 shutdown_connections 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.335+0000 7fe4b53e9640 1 -- 172.21.15.162:0/376442750 wait complete. 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.336+0000 7fe4b53e9640 10 monclient: build_initial_monmap 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.336+0000 7fe4b53e9640 10 monclient: monmap: 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.336+0000 7fe4b53e9640 1 librados: starting msgr at 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.336+0000 7fe4b53e9640 1 librados: starting objecter 2024-01-27T03:42:27.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 Processor -- start 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 -- start start 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 librados: setting wanted keys 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 librados: calling monclient init 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient: init 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b00b3db0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b00b4b00 0x7fe4b013b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b00b4b00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b013f730 0x7fe4b013c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b013f730 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b013d440 con 0x7fe4b00b4b00 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b013bac0 con 0x7fe4b013f730 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b013d6a0 con 0x7fe4b00b3db0 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.337+0000 7fe4b53e9640 10 monclient(hunting): authenticate will time out at 1439.681568s 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.338+0000 7fe4ae7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b00b4b00 0x7fe4b013b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.338+0000 7fe4ae7fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b00b4b00 0x7fe4b013b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.162:54034/0 (socket says 172.21.15.162:54034) 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.338+0000 7fe4ae7fc640 1 -- 172.21.15.162:0/2292770610 learned_addr learned my addr 172.21.15.162:0/2292770610 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.338+0000 7fe4ae7fc640 10 monclient(hunting): get_auth_request con 0x7fe4b00b4b00 auth_method 0 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.338+0000 7fe4ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.338+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.338+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.339+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.339+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.339+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.342+0000 7fe4aeffd640 1 --2- 172.21.15.162:0/2292770610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_done global_id 4116 payload 824 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 1 -- 172.21.15.162:0/2292770610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 msgr2=0x7fe4b0138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 1 --2- 172.21.15.162:0/2292770610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0138c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 1 -- 172.21.15.162:0/2292770610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b013f730 msgr2=0x7fe4b013c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 1 --2- 172.21.15.162:0/2292770610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b013f730 0x7fe4b013c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 10 monclient: _finish_hunting 0 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 1 monclient: found mon.noname-a 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4ae7fc640 1 -- 172.21.15.162:0/2292770610 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a8009a70 con 0x7fe4b00b4b00 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4aeffd640 10 monclient: get_auth_request con 0x7fe4b00b3db0 auth_method 0 2024-01-27T03:42:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.343+0000 7fe4aeffd640 1 --2- 172.21.15.162:0/2292770610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0138c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-27T03:42:27.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.345+0000 7fe4ae7fc640 1 --2- 172.21.15.162:0/2292770610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b00b4b00 0x7fe4b013b580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000c390 tx=0x7fe4a000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:27.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.345+0000 7fe4977fe640 10 client.?.objecter ms_handle_connect 0x7fe4b00b4b00 2024-01-27T03:42:27.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.345+0000 7fe4977fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 1 -- 172.21.15.162:0/2292770610 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0005ce0 con 0x7fe4b00b4b00 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 10 monclient: dump: 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.350 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 10 monclient: _finish_auth 0 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 10 monclient: _check_auth_tickets 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4b53e9640 5 monclient: authenticate success, global_id 4116 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 1 -- 172.21.15.162:0/2292770610 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a0005e40 con 0x7fe4b00b4b00 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 1 -- 172.21.15.162:0/2292770610 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00072f0 con 0x7fe4b00b4b00 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4977fe640 10 monclient: dump: 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:27.351 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4ad7fa640 4 set_mon_vals no callback set 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4b53e9640 10 monclient: _renew_subs 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4b53e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4b53e9640 1 -- 172.21.15.162:0/2292770610 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b013bce0 con 0x7fe4b00b4b00 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4b53e9640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4b53e9640 10 monclient: _renew_subs 2024-01-27T03:42:27.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4b53e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:27.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4b53e9640 1 -- 172.21.15.162:0/2292770610 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b01db0c0 con 0x7fe4b00b4b00 2024-01-27T03:42:27.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.349+0000 7fe4b53e9640 1 librados: init done 2024-01-27T03:42:27.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.350+0000 7fe4b53e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:27.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.350+0000 7fe4b53e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:27.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.350+0000 7fe4b53e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:27.353 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.350+0000 7fe4b53e9640 1 -- 172.21.15.162:0/2292770610 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b013da40 con 0x7fe4b00b4b00 2024-01-27T03:42:27.362 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.362+0000 7f8733841040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T03:42:27.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.366+0000 7fe4977fe640 1 -- 172.21.15.162:0/2292770610 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe4a0007450 con 0x7fe4b00b4b00 2024-01-27T03:42:27.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.366+0000 7fe4977fe640 1 -- 172.21.15.162:0/2292770610 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe4a0012740 con 0x7fe4b00b4b00 2024-01-27T03:42:27.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.366+0000 7fe4977fe640 10 client.4116.objecter ms_dispatch 0x7fe4b0000b90 osd_map(2..2 src has 1..2) v4 2024-01-27T03:42:27.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.366+0000 7fe4977fe640 3 client.4116.objecter handle_osd_map got epochs [2,2] > 0 2024-01-27T03:42:27.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.366+0000 7fe4977fe640 3 client.4116.objecter handle_osd_map decoding full epoch 2 2024-01-27T03:42:27.367 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.366+0000 7fe4977fe640 20 client.4116.objecter dump_active .. 0 homeless 2024-01-27T03:42:27.417 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.417+0000 7f8733841040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T03:42:27.462 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.462+0000 7f8733841040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T03:42:27.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.481+0000 7fe4977fe640 1 -- 172.21.15.162:0/2292770610 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7fe4b013da40 con 0x7fe4b00b4b00 2024-01-27T03:42:27.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.481+0000 7fe4977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:27.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.481+0000 7fe4977fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:27.517 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.517+0000 7f8733841040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T03:42:27.543 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:27.542+0000 7f1407c0f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-27T03:42:27.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.555+0000 7fe4b53e9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "112d1665-ee1b-41ea-85fc-4165aa26818e", "id": 0}] 2024-01-27T03:42:27.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.555+0000 7fe4b53e9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "112d1665-ee1b-41ea-85fc-4165aa26818e", "id": 0}] 2024-01-27T03:42:27.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.555+0000 7fe4b53e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:27.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.555+0000 7fe4b53e9640 1 -- 172.21.15.162:0/2292770610 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "112d1665-ee1b-41ea-85fc-4165aa26818e", "id": 0} v 0) v1 -- 0x7fe4b00b3b50 con 0x7fe4b00b4b00 2024-01-27T03:42:27.560 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.560+0000 7f8733841040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T03:42:27.598 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:27.597+0000 7f1407c0f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-27T03:42:27.662 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.662+0000 7f8733841040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T03:42:27.664 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:27.663+0000 7f1407c0f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-27T03:42:27.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.714+0000 7fe4977fe640 1 -- 172.21.15.162:0/2292770610 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "112d1665-ee1b-41ea-85fc-4165aa26818e", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe4a0017070 con 0x7fe4b00b4b00 2024-01-27T03:42:27.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.714+0000 7fe4977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "112d1665-ee1b-41ea-85fc-4165aa26818e", "id": 0}] 2024-01-27T03:42:27.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.714+0000 7fe4977fe640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:27.715 INFO:teuthology.orchestra.run.smithi162.stdout:0 2024-01-27T03:42:27.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 10 librados: watch_flush enter 2024-01-27T03:42:27.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 10 librados: watch_flush exit 2024-01-27T03:42:27.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 20 client.4116.objecter shutdown clearing up homeless session... 2024-01-27T03:42:27.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 10 client.4116.objecter successfully canceled tick 2024-01-27T03:42:27.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 10 monclient: shutdown 2024-01-27T03:42:27.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:27.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 -- 172.21.15.162:0/2292770610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b00b4b00 msgr2=0x7fe4b013b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:27.717 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 --2- 172.21.15.162:0/2292770610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b00b4b00 0x7fe4b013b580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000c390 tx=0x7fe4a000c850 comp rx=0 tx=0).stop 2024-01-27T03:42:27.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 -- 172.21.15.162:0/2292770610 shutdown_connections 2024-01-27T03:42:27.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 --2- 172.21.15.162:0/2292770610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b013f730 0x7fe4b013c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 --2- 172.21.15.162:0/2292770610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b00b4b00 0x7fe4b013b580 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 --2- 172.21.15.162:0/2292770610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b00b3db0 0x7fe4b0138c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 -- 172.21.15.162:0/2292770610 >> 172.21.15.162:0/2292770610 conn(0x7fe4b0066490 msgr2=0x7fe4b00b2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:27.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 -- 172.21.15.162:0/2292770610 shutdown_connections 2024-01-27T03:42:27.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 -- 172.21.15.162:0/2292770610 wait complete. 2024-01-27T03:42:27.718 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.715+0000 7fe4b53e9640 1 librados: shutdown 2024-01-27T03:42:27.722 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:27.721+0000 7f1407c0f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-27T03:42:27.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 1c7f3c75-90ac-4a98-bc89-b6bd97b20f49 1 2024-01-27T03:42:27.745 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.745+0000 7f8733841040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T03:42:27.766 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:27.766+0000 7f1407c0f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-27T03:42:27.795 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.795+0000 7f8733841040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T03:42:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.826+0000 7f389bd9e640 10 monclient: get_monmap_and_config 2024-01-27T03:42:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.826+0000 7f389bd9e640 10 monclient: build_initial_monmap 2024-01-27T03:42:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.826+0000 7f389bd9e640 10 monclient: monmap: 2024-01-27T03:42:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:42:27.827401+0000 2024-01-27T03:42:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:42:27.827401+0000 2024-01-27T03:42:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.828 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:27.828 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:27.828 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:27.828 INFO:teuthology.orchestra.run.smithi162.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:27.828 INFO:teuthology.orchestra.run.smithi162.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:27.828 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.827+0000 7f389bd9e640 1 Processor -- start 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.827+0000 7f389bd9e640 1 -- start start 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.827+0000 7f389bd9e640 10 monclient: init 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f38940b4680 0x7f3894142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient(hunting): picked mon.noname-c con 0x7f38940b4680 addr v2:172.21.15.162:3300/0 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4a60 0x7f38940b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient(hunting): picked mon.noname-b con 0x7f38940b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3894142eb0 0x7f38940b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient(hunting): picked mon.noname-a con 0x7f3894142eb0 addr v2:172.21.15.110:3300/0 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3894142240 con 0x7f3894142eb0 2024-01-27T03:42:27.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38940afbe0 con 0x7f38940b4680 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38940b0890 con 0x7f38940b4a60 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.828+0000 7f389bd9e640 10 monclient(hunting): authenticate will time out at 1440.172259s 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3894142eb0 0x7f38940b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3894142eb0 0x7f38940b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.162:54044/0 (socket says 172.21.15.162:54044) 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 1 -- 172.21.15.162:0/1710005498 learned_addr learned my addr 172.21.15.162:0/1710005498 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 10 monclient(hunting): get_auth_request con 0x7f3894142eb0 auth_method 0 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:27.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:27.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.829+0000 7f389a33b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:27.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 10 monclient(hunting): handle_auth_done global_id 4119 payload 274 2024-01-27T03:42:27.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 1 -- 172.21.15.162:0/1710005498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4a60 msgr2=0x7f38940b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:27.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 1 --2- 172.21.15.162:0/1710005498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4a60 0x7f38940b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 1 -- 172.21.15.162:0/1710005498 >> v2:172.21.15.162:3300/0 conn(0x7f38940b4680 msgr2=0x7f3894142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:27.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 1 --2- 172.21.15.162:0/1710005498 >> v2:172.21.15.162:3300/0 conn(0x7f38940b4680 0x7f3894142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 10 monclient: _finish_hunting 0 2024-01-27T03:42:27.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 1 monclient: found mon.noname-a 2024-01-27T03:42:27.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:27.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:27.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.831+0000 7f389a33b640 1 -- 172.21.15.162:0/1710005498 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38940b0b00 con 0x7f3894142eb0 2024-01-27T03:42:27.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.841+0000 7f389a33b640 1 --2- 172.21.15.162:0/1710005498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894142eb0 0x7f38940b19b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f38940b4ee0 tx=0x7f388c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4f4183db704065 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:27.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 1 -- 172.21.15.162:0/1710005498 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c02c9e0 con 0x7f3894142eb0 2024-01-27T03:42:27.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:27.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:27.845 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 10 monclient: dump: 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 10 monclient: _finish_auth 0 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 10 monclient: _check_auth_tickets 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:27.846 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 1 -- 172.21.15.162:0/1710005498 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388c02cb40 con 0x7f3894142eb0 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 1 -- 172.21.15.162:0/1710005498 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c0343f0 con 0x7f3894142eb0 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f389bd9e640 5 monclient: authenticate success, global_id 4119 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f389bd9e640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f389bd9e640 4 set_mon_vals no callback set 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f3898b38640 10 monclient: dump: 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:27.848 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:27.848 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:27.848 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f389bd9e640 10 monclient: shutdown 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f389bd9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f389bd9e640 1 -- 172.21.15.162:0/1710005498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894142eb0 msgr2=0x7f38940b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.845+0000 7f389bd9e640 1 --2- 172.21.15.162:0/1710005498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894142eb0 0x7f38940b19b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f38940b4ee0 tx=0x7f388c02be40 comp rx=0 tx=0).stop 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.846+0000 7f389bd9e640 1 -- 172.21.15.162:0/1710005498 shutdown_connections 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.846+0000 7f389bd9e640 1 --2- 172.21.15.162:0/1710005498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894142eb0 0x7f38940b19b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.846+0000 7f389bd9e640 1 --2- 172.21.15.162:0/1710005498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4a60 0x7f38940b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.846+0000 7f389bd9e640 1 --2- 172.21.15.162:0/1710005498 >> v2:172.21.15.162:3300/0 conn(0x7f38940b4680 0x7f3894142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.846+0000 7f389bd9e640 1 -- 172.21.15.162:0/1710005498 >> 172.21.15.162:0/1710005498 conn(0x7f3894066490 msgr2=0x7f38940668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.846+0000 7f389bd9e640 1 -- 172.21.15.162:0/1710005498 shutdown_connections 2024-01-27T03:42:27.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.846+0000 7f389bd9e640 1 -- 172.21.15.162:0/1710005498 wait complete. 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.846+0000 7f389bd9e640 10 monclient: build_initial_monmap 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.846+0000 7f389bd9e640 10 monclient: monmap: 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 librados: starting msgr at 2024-01-27T03:42:27.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 librados: starting objecter 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 Processor -- start 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 -- start start 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 librados: setting wanted keys 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 librados: calling monclient init 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient: init 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4680 0x7f38940a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient(hunting): picked mon.noname-c con 0x7f38940b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3894142eb0 0x7f38941db110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient(hunting): picked mon.noname-b con 0x7f3894142eb0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940aa180 0x7f38941db650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient(hunting): picked mon.noname-a con 0x7f38940aa180 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.853 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38941cef90 con 0x7f38940aa180 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38940ab250 con 0x7f3894142eb0 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38941dbb90 con 0x7f38940b4680 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.847+0000 7f389bd9e640 10 monclient(hunting): authenticate will time out at 1440.191681s 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f389a33b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940aa180 0x7f38941db650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f3899b3a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4680 0x7f38940a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f3899b3a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4680 0x7f38940a9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.162:49638/0 (socket says 172.21.15.162:49638) 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f3899b3a640 1 -- 172.21.15.162:0/3868494841 learned_addr learned my addr 172.21.15.162:0/3868494841 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f3899b3a640 10 monclient(hunting): get_auth_request con 0x7f38940b4680 auth_method 0 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f3899b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f3899b3a640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:27.854 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f3899b3a640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f389a33b640 10 monclient(hunting): get_auth_request con 0x7f38940aa180 auth_method 0 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f389a33b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f389a33b640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.848+0000 7f389a33b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.849+0000 7f3899b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.849+0000 7f3899b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.849+0000 7f3899b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.849+0000 7f389a33b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.849+0000 7f389a33b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:27.855 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.849+0000 7f389a33b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:27.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 10 monclient(hunting): handle_auth_done global_id 4112 payload 824 2024-01-27T03:42:27.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 1 -- 172.21.15.162:0/3868494841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3894142eb0 msgr2=0x7f38941db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:27.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 1 --2- 172.21.15.162:0/3868494841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3894142eb0 0x7f38941db110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 1 -- 172.21.15.162:0/3868494841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940aa180 msgr2=0x7f38941db650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:27.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 1 --2- 172.21.15.162:0/3868494841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940aa180 0x7f38941db650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:27.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 10 monclient: _finish_hunting 0 2024-01-27T03:42:27.856 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 1 monclient: found mon.noname-c 2024-01-27T03:42:27.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:27.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:42:27.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f3899b3a640 1 -- 172.21.15.162:0/3868494841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c009b30 con 0x7f38940b4680 2024-01-27T03:42:27.857 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.851+0000 7f389a33b640 1 --2- 172.21.15.162:0/3868494841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940aa180 0x7f38941db650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T03:42:27.857 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.853+0000 7f8733841040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T03:42:27.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.854+0000 7f3899b3a640 1 --2- 172.21.15.162:0/3868494841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4680 0x7f38940a9b90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f389000b4d0 tx=0x7f389000b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:27.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.854+0000 7f388a7fc640 10 client.?.objecter ms_handle_connect 0x7f38940b4680 2024-01-27T03:42:27.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.854+0000 7f388a7fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:27.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 1 -- 172.21.15.162:0/3868494841 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890010870 con 0x7f38940b4680 2024-01-27T03:42:27.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:27.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:42:27.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 10 monclient: dump: 2024-01-27T03:42:27.859 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 10 monclient: _finish_auth 0 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:42:27.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 1 -- 172.21.15.162:0/3868494841 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38900109d0 con 0x7f38940b4680 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 1 -- 172.21.15.162:0/3868494841 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890010d80 con 0x7f38940b4680 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f389bd9e640 5 monclient: authenticate success, global_id 4112 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f389bd9e640 10 monclient: _renew_subs 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f389bd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f389bd9e640 1 -- 172.21.15.162:0/3868494841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38941dbdb0 con 0x7f38940b4680 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388bfff640 4 set_mon_vals no callback set 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f388a7fc640 10 monclient: dump: 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:27.861 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f389bd9e640 10 client.4112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f389bd9e640 10 monclient: _renew_subs 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f389bd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f389bd9e640 1 -- 172.21.15.162:0/3868494841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38941cc720 con 0x7f38940b4680 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.857+0000 7f389bd9e640 1 librados: init done 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.859+0000 7f389bd9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.859+0000 7f389bd9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:27.862 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.859+0000 7f389bd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:27.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:27.859+0000 7f389bd9e640 1 -- 172.21.15.162:0/3868494841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38940b0260 con 0x7f38940b4680 2024-01-27T03:42:27.887 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:27.887+0000 7f1407c0f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-27T03:42:27.902 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:27.902+0000 7f8733841040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T03:42:27.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:27.959+0000 7f1407c0f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-27T03:42:28.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:28.006+0000 7f1407c0f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-27T03:42:28.033 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.032+0000 7f388a7fc640 1 -- 172.21.15.162:0/3868494841 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3890023480 con 0x7f38940b4680 2024-01-27T03:42:28.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.039+0000 7f388a7fc640 1 -- 172.21.15.162:0/3868494841 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f3890019b50 con 0x7f38940b4680 2024-01-27T03:42:28.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.039+0000 7f388a7fc640 10 client.4112.objecter ms_dispatch 0x7f3894000b90 osd_map(3..3 src has 1..3) v4 2024-01-27T03:42:28.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.039+0000 7f388a7fc640 3 client.4112.objecter handle_osd_map got epochs [3,3] > 0 2024-01-27T03:42:28.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.039+0000 7f388a7fc640 3 client.4112.objecter handle_osd_map decoding full epoch 3 2024-01-27T03:42:28.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.039+0000 7f388a7fc640 20 client.4112.objecter dump_active .. 0 homeless 2024-01-27T03:42:28.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.039+0000 7f388a7fc640 1 -- 172.21.15.162:0/3868494841 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7f389001e060 con 0x7f38940b4680 2024-01-27T03:42:28.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.039+0000 7f388a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:28.040 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.039+0000 7f388a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:28.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:28.061+0000 7f1407c0f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-27T03:42:28.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:28.114+0000 7f1407c0f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-27T03:42:28.115 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.115+0000 7f389bd9e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1c7f3c75-90ac-4a98-bc89-b6bd97b20f49", "id": 1}] 2024-01-27T03:42:28.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.115+0000 7f389bd9e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1c7f3c75-90ac-4a98-bc89-b6bd97b20f49", "id": 1}] 2024-01-27T03:42:28.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.115+0000 7f389bd9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:28.116 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.115+0000 7f389bd9e640 1 -- 172.21.15.162:0/3868494841 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1c7f3c75-90ac-4a98-bc89-b6bd97b20f49", "id": 1} v 0) v1 -- 0x7f389404ea50 con 0x7f38940b4680 2024-01-27T03:42:28.292 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:28.292+0000 7f8733841040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T03:42:28.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.302+0000 7f388a7fc640 1 -- 172.21.15.162:0/3868494841 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1c7f3c75-90ac-4a98-bc89-b6bd97b20f49", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3890027020 con 0x7f38940b4680 2024-01-27T03:42:28.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.302+0000 7f388a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1c7f3c75-90ac-4a98-bc89-b6bd97b20f49", "id": 1}] 2024-01-27T03:42:28.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.302+0000 7f388a7fc640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:28.303 INFO:teuthology.orchestra.run.smithi162.stdout:1 2024-01-27T03:42:28.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 10 librados: watch_flush enter 2024-01-27T03:42:28.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 10 librados: watch_flush exit 2024-01-27T03:42:28.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 20 client.4112.objecter shutdown clearing up homeless session... 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 10 client.4112.objecter successfully canceled tick 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 10 monclient: shutdown 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 -- 172.21.15.162:0/3868494841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4680 msgr2=0x7f38940a9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 --2- 172.21.15.162:0/3868494841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4680 0x7f38940a9b90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f389000b4d0 tx=0x7f389000b990 comp rx=0 tx=0).stop 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 -- 172.21.15.162:0/3868494841 shutdown_connections 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 --2- 172.21.15.162:0/3868494841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940aa180 0x7f38941db650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 --2- 172.21.15.162:0/3868494841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3894142eb0 0x7f38941db110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 --2- 172.21.15.162:0/3868494841 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38940b4680 0x7f38940a9b90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 -- 172.21.15.162:0/3868494841 >> 172.21.15.162:0/3868494841 conn(0x7f3894066490 msgr2=0x7f389413e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 -- 172.21.15.162:0/3868494841 shutdown_connections 2024-01-27T03:42:28.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 -- 172.21.15.162:0/3868494841 wait complete. 2024-01-27T03:42:28.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.303+0000 7f389bd9e640 1 librados: shutdown 2024-01-27T03:42:28.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 5d9d24ee-f76a-47e1-86c8-6ec8e22805bb 2 2024-01-27T03:42:28.433 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:28.432+0000 7f8733841040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T03:42:28.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.454+0000 7f1e1f905640 10 monclient: get_monmap_and_config 2024-01-27T03:42:28.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.454+0000 7f1e1f905640 10 monclient: build_initial_monmap 2024-01-27T03:42:28.455 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.454+0000 7f1e1f905640 10 monclient: monmap: 2024-01-27T03:42:28.455 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:28.455 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.455 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:42:28.454795+0000 2024-01-27T03:42:28.455 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:42:28.454795+0000 2024-01-27T03:42:28.455 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:28.455 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.455 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:28.456 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:28.456 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:28.456 INFO:teuthology.orchestra.run.smithi162.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:28.456 INFO:teuthology.orchestra.run.smithi162.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:28.456 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.456 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.454+0000 7f1e1f905640 1 Processor -- start 2024-01-27T03:42:28.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 1 -- start start 2024-01-27T03:42:28.458 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 10 monclient: init 2024-01-27T03:42:28.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:28.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-01-27T03:42:28.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 0x7f1e18142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:28.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 10 monclient(hunting): picked mon.noname-b con 0x7f1e180b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:28.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e180b4a60 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:28.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e18142eb0 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:28.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e180afbe0 con 0x7f1e180b4a60 2024-01-27T03:42:28.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e180b0890 con 0x7f1e18142eb0 2024-01-27T03:42:28.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e18142240 con 0x7f1e180b4680 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.455+0000 7f1e1f905640 10 monclient(hunting): authenticate will time out at 1440.799570s 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.456+0000 7f1e1d6a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 0x7f1e18142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.456+0000 7f1e1cea0640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1e180b4a60 0x7f1e180b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.162:46546/0 (socket says 172.21.15.162:46546) 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.456+0000 7f1e1cea0640 1 -- 172.21.15.162:0/346324115 learned_addr learned my addr 172.21.15.162:0/346324115 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.457+0000 7f1e1d6a1640 10 monclient(hunting): get_auth_request con 0x7f1e180b4680 auth_method 0 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.457+0000 7f1e1d6a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.457+0000 7f1e1d6a1640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:28.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.457+0000 7f1e1d6a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:28.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.457+0000 7f1e1d6a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:28.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.457+0000 7f1e1d6a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:28.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.457+0000 7f1e1d6a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:28.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e0ffff640 1 -- 172.21.15.162:0/346324115 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953154867 0 0) 0x7f1e180afbe0 con 0x7f1e180b4a60 2024-01-27T03:42:28.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e0ffff640 10 monclient(hunting): _negotiate 2024-01-27T03:42:28.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e0ffff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:28.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:28.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e0ffff640 10 monclient(hunting): my global_id is 4125 2024-01-27T03:42:28.462 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e0ffff640 1 -- 172.21.15.162:0/346324115 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e04003520 con 0x7f1e180b4a60 2024-01-27T03:42:28.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e1d6a1640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-01-27T03:42:28.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e1d6a1640 1 -- 172.21.15.162:0/346324115 >> v1:172.21.15.162:6789/0 conn(0x7f1e18142eb0 legacy=0x7f1e1814caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e1d6a1640 1 -- 172.21.15.162:0/346324115 >> v1:172.21.15.110:6789/0 conn(0x7f1e180b4a60 legacy=0x7f1e180b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e1d6a1640 10 monclient: _finish_hunting 0 2024-01-27T03:42:28.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e1d6a1640 1 monclient: found mon.noname-b 2024-01-27T03:42:28.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e1d6a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:28.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e1d6a1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:42:28.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.459+0000 7f1e1d6a1640 1 -- 172.21.15.162:0/346324115 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e180b0b00 con 0x7f1e180b4680 2024-01-27T03:42:28.463 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.461+0000 7f1e1d6a1640 1 --2- 172.21.15.162:0/346324115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 0x7f1e18142970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1e080097d0 tx=0x7f1e0802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=405d402852ca4c2e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 1 -- 172.21.15.162:0/346324115 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e08031800 con 0x7f1e180b4680 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 10 monclient: dump: 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.466 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 10 monclient: _finish_auth 0 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 10 monclient: _check_auth_tickets 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e1f905640 5 monclient: authenticate success, global_id 4115 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 1 -- 172.21.15.162:0/346324115 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e08031960 con 0x7f1e180b4680 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e1f905640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e1f905640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 1 -- 172.21.15.162:0/346324115 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e08031cf0 con 0x7f1e180b4680 2024-01-27T03:42:28.467 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e1f905640 4 set_mon_vals no callback set 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.465+0000 7f1e0ffff640 10 monclient: dump: 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.468 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:28.469 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:28.469 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:28.469 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.466+0000 7f1e1f905640 10 monclient: shutdown 2024-01-27T03:42:28.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.466+0000 7f1e1f905640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:28.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.466+0000 7f1e1f905640 1 -- 172.21.15.162:0/346324115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 msgr2=0x7f1e18142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.466+0000 7f1e1f905640 1 --2- 172.21.15.162:0/346324115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 0x7f1e18142970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1e080097d0 tx=0x7f1e0802b8a0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.473 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.466+0000 7f1e1f905640 1 -- 172.21.15.162:0/346324115 shutdown_connections 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.466+0000 7f1e1f905640 1 --2- 172.21.15.162:0/346324115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 0x7f1e18142970 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.466+0000 7f1e1f905640 1 -- 172.21.15.162:0/346324115 >> 172.21.15.162:0/346324115 conn(0x7f1e18066490 msgr2=0x7f1e180668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.466+0000 7f1e1f905640 1 -- 172.21.15.162:0/346324115 shutdown_connections 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.466+0000 7f1e1f905640 1 -- 172.21.15.162:0/346324115 wait complete. 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.467+0000 7f1e1f905640 10 monclient: build_initial_monmap 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.467+0000 7f1e1f905640 10 monclient: monmap: 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.474 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:28.475 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:28.475 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:28.475 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.467+0000 7f1e1f905640 1 librados: starting msgr at 2024-01-27T03:42:28.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.467+0000 7f1e1f905640 1 librados: starting objecter 2024-01-27T03:42:28.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.467+0000 7f1e1f905640 1 Processor -- start 2024-01-27T03:42:28.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.467+0000 7f1e1f905640 1 -- start start 2024-01-27T03:42:28.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.467+0000 7f1e1f905640 1 librados: setting wanted keys 2024-01-27T03:42:28.475 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.467+0000 7f1e1f905640 1 librados: calling monclient init 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.467+0000 7f1e1f905640 10 monclient: init 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 0x7f1e18145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient(hunting): picked mon.noname-c con 0x7f1e180b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e18142eb0 0x7f1e180ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient(hunting): picked mon.noname-a con 0x7f1e18142eb0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180aaf20 0x7f1e180ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient(hunting): picked mon.noname-b con 0x7f1e180aaf20 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180abff0 con 0x7f1e18142eb0 2024-01-27T03:42:28.476 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180ad150 con 0x7f1e180aaf20 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180abb60 con 0x7f1e180b4680 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1f905640 10 monclient(hunting): authenticate will time out at 1440.812037s 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1d6a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 0x7f1e18145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.468+0000 7f1e1cea0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e18142eb0 0x7f1e180ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1cea0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e18142eb0 0x7f1e180ab400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.162:54066/0 (socket says 172.21.15.162:54066) 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1cea0640 1 -- 172.21.15.162:0/4123190572 learned_addr learned my addr 172.21.15.162:0/4123190572 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1cea0640 10 monclient(hunting): get_auth_request con 0x7f1e18142eb0 auth_method 0 2024-01-27T03:42:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1cea0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1cea0640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1cea0640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1d6a1640 10 monclient(hunting): get_auth_request con 0x7f1e180b4680 auth_method 0 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1d6a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1d6a1640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1d6a1640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1cea0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1cea0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1cea0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1d6a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:28.478 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1d6a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.469+0000 7f1e1d6a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 10 monclient(hunting): handle_auth_done global_id 4128 payload 824 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 1 -- 172.21.15.162:0/4123190572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 msgr2=0x7f1e18145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 1 --2- 172.21.15.162:0/4123190572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 0x7f1e18145610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 1 -- 172.21.15.162:0/4123190572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180aaf20 msgr2=0x7f1e180ad4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 1 --2- 172.21.15.162:0/4123190572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180aaf20 0x7f1e180ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 10 monclient: _finish_hunting 0 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 1 monclient: found mon.noname-a 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.479 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.472+0000 7f1e1cea0640 1 -- 172.21.15.162:0/4123190572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e08009400 con 0x7f1e18142eb0 2024-01-27T03:42:28.481 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.474+0000 7f1e1cea0640 1 --2- 172.21.15.162:0/4123190572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e18142eb0 0x7f1e180ab400 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400c500 tx=0x7f1e1400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.474+0000 7f1e0dffb640 10 client.?.objecter ms_handle_connect 0x7f1e18142eb0 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.474+0000 7f1e0dffb640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.477+0000 7f1e0dffb640 1 -- 172.21.15.162:0/4123190572 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e14005ce0 con 0x7f1e18142eb0 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 10 monclient: dump: 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:28.482 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 10 monclient: _finish_auth 0 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 10 monclient: _check_auth_tickets 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 1 -- 172.21.15.162:0/4123190572 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e14005e40 con 0x7f1e18142eb0 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e1f905640 5 monclient: authenticate success, global_id 4128 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e1f905640 10 monclient: _renew_subs 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e1f905640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.483 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e1f905640 1 -- 172.21.15.162:0/4123190572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e180adae0 con 0x7f1e18142eb0 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 1 -- 172.21.15.162:0/4123190572 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e140072f0 con 0x7f1e18142eb0 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0dffb640 10 monclient: dump: 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.484 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e0f7fe640 4 set_mon_vals no callback set 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e1f905640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e1f905640 10 monclient: _renew_subs 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e1f905640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e1f905640 1 -- 172.21.15.162:0/4123190572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e180adff0 con 0x7f1e18142eb0 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.478+0000 7f1e1f905640 1 librados: init done 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.479+0000 7f1e1f905640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.479+0000 7f1e1f905640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:28.485 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.479+0000 7f1e1f905640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.479+0000 7f1e1f905640 1 -- 172.21.15.162:0/4123190572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e18074d10 con 0x7f1e18142eb0 2024-01-27T03:42:28.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.480+0000 7f1e0dffb640 1 -- 172.21.15.162:0/4123190572 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1e14007510 con 0x7f1e18142eb0 2024-01-27T03:42:28.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.480+0000 7f1e0dffb640 1 -- 172.21.15.162:0/4123190572 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f1e1401b7d0 con 0x7f1e18142eb0 2024-01-27T03:42:28.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.480+0000 7f1e0dffb640 10 client.4128.objecter ms_dispatch 0x7f1e18000b90 osd_map(4..4 src has 1..4) v4 2024-01-27T03:42:28.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.480+0000 7f1e0dffb640 3 client.4128.objecter handle_osd_map got epochs [4,4] > 0 2024-01-27T03:42:28.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.480+0000 7f1e0dffb640 3 client.4128.objecter handle_osd_map decoding full epoch 4 2024-01-27T03:42:28.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.480+0000 7f1e0dffb640 20 client.4128.objecter dump_active .. 0 homeless 2024-01-27T03:42:28.550 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:28.549+0000 7f1407c0f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-27T03:42:28.560 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:28.560+0000 7f8733841040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T03:42:28.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.578+0000 7f1e0dffb640 1 -- 172.21.15.162:0/4123190572 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7f1e14021060 con 0x7f1e18142eb0 2024-01-27T03:42:28.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.579+0000 7f1e0dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:28.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.579+0000 7f1e0dffb640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:28.606 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:28.606+0000 7f8733841040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T03:42:28.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.651+0000 7f1e1f905640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5d9d24ee-f76a-47e1-86c8-6ec8e22805bb", "id": 2}] 2024-01-27T03:42:28.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.651+0000 7f1e1f905640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5d9d24ee-f76a-47e1-86c8-6ec8e22805bb", "id": 2}] 2024-01-27T03:42:28.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.651+0000 7f1e1f905640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.651+0000 7f1e1f905640 1 -- 172.21.15.162:0/4123190572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5d9d24ee-f76a-47e1-86c8-6ec8e22805bb", "id": 2} v 0) v1 -- 0x7f1e180abca0 con 0x7f1e18142eb0 2024-01-27T03:42:28.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.680+0000 7f1e0dffb640 1 -- 172.21.15.162:0/4123190572 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5d9d24ee-f76a-47e1-86c8-6ec8e22805bb", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1e14017070 con 0x7f1e18142eb0 2024-01-27T03:42:28.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.680+0000 7f1e0dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5d9d24ee-f76a-47e1-86c8-6ec8e22805bb", "id": 2}] 2024-01-27T03:42:28.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.680+0000 7f1e0dffb640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:28.681 INFO:teuthology.orchestra.run.smithi162.stdout:2 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 10 librados: watch_flush enter 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 10 librados: watch_flush exit 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 20 client.4128.objecter shutdown clearing up homeless session... 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 10 client.4128.objecter successfully canceled tick 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 10 monclient: shutdown 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 -- 172.21.15.162:0/4123190572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e18142eb0 msgr2=0x7f1e180ab400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 --2- 172.21.15.162:0/4123190572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e18142eb0 0x7f1e180ab400 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400c500 tx=0x7f1e1400c9c0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 -- 172.21.15.162:0/4123190572 shutdown_connections 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 --2- 172.21.15.162:0/4123190572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180aaf20 0x7f1e180ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 --2- 172.21.15.162:0/4123190572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e18142eb0 0x7f1e180ab400 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 --2- 172.21.15.162:0/4123190572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e180b4680 0x7f1e18145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 -- 172.21.15.162:0/4123190572 >> 172.21.15.162:0/4123190572 conn(0x7f1e18066490 msgr2=0x7f1e180b73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:28.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 -- 172.21.15.162:0/4123190572 shutdown_connections 2024-01-27T03:42:28.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 -- 172.21.15.162:0/4123190572 wait complete. 2024-01-27T03:42:28.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.682+0000 7f1e1f905640 1 librados: shutdown 2024-01-27T03:42:28.688 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:28.687+0000 7f1407c0f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-27T03:42:28.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 2b5190b6-abce-4528-9ea6-8363bdd89328 3 2024-01-27T03:42:28.707 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:28.707+0000 7f8733841040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T03:42:28.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.796+0000 7f0ae4f3e640 10 monclient: get_monmap_and_config 2024-01-27T03:42:28.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.796+0000 7f0ae4f3e640 10 monclient: build_initial_monmap 2024-01-27T03:42:28.797 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.796+0000 7f0ae4f3e640 10 monclient: monmap: 2024-01-27T03:42:28.797 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:28.797 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:42:28.797148+0000 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:42:28.797148+0000 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.797+0000 7f0ae4f3e640 1 Processor -- start 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.797+0000 7f0ae4f3e640 1 -- start start 2024-01-27T03:42:28.798 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.797+0000 7f0ae4f3e640 10 monclient: init 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.797+0000 7f0ae4f3e640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.797+0000 7f0ae4f3e640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae00b4680 0x7f0ae0142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae00b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0ae00b4a60 0x7f0ae00b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae00b4a60 addr v2:172.21.15.162:3300/0 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ae0142eb0 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae00afbe0 con 0x7f0ae0142eb0 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae0142240 con 0x7f0ae00b4a60 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:28.801 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae00b0890 con 0x7f0ae00b4680 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ae4f3e640 10 monclient(hunting): authenticate will time out at 1441.141924s 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0addd9b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0ae00b4a60 0x7f0ae00b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.798+0000 7f0ade59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae00b4680 0x7f0ae0142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0addd9b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0ae00b4a60 0x7f0ae00b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:49460/0 (socket says 172.21.15.162:49460) 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0ade59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae00b4680 0x7f0ae0142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.162:49674/0 (socket says 172.21.15.162:49674) 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0addd9b640 1 -- 172.21.15.162:0/3439866804 learned_addr learned my addr 172.21.15.162:0/3439866804 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0addd9b640 10 monclient(hunting): get_auth_request con 0x7f0ae00b4a60 auth_method 0 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0addd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0addd9b640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0addd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0ade59c640 10 monclient(hunting): get_auth_request con 0x7f0ae00b4680 auth_method 0 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0ade59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0ade59c640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:28.802 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0ade59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:28.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0ade59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:28.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0ade59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:28.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0ade59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:28.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0addd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:28.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0addd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:28.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.799+0000 7f0addd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:28.803 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.801+0000 7f0ade59c640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-01-27T03:42:28.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.801+0000 7f0ade59c640 1 -- 172.21.15.162:0/3439866804 >> v2:172.21.15.162:3300/0 conn(0x7f0ae00b4a60 msgr2=0x7f0ae00b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:28.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.801+0000 7f0ade59c640 1 --2- 172.21.15.162:0/3439866804 >> v2:172.21.15.162:3300/0 conn(0x7f0ae00b4a60 0x7f0ae00b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.801+0000 7f0ade59c640 1 -- 172.21.15.162:0/3439866804 >> v1:172.21.15.110:6789/0 conn(0x7f0ae0142eb0 legacy=0x7f0ae00b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.801+0000 7f0ade59c640 10 monclient: _finish_hunting 0 2024-01-27T03:42:28.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.801+0000 7f0ade59c640 1 monclient: found mon.noname-b 2024-01-27T03:42:28.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.801+0000 7f0ade59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:28.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.801+0000 7f0ade59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:42:28.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.801+0000 7f0ade59c640 1 -- 172.21.15.162:0/3439866804 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae00b0b00 con 0x7f0ae00b4680 2024-01-27T03:42:28.804 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.803+0000 7f0ade59c640 1 --2- 172.21.15.162:0/3439866804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae00b4680 0x7f0ae0142970 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00097a0 tx=0x7f0ad002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=608f824f592bc320 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:28.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.806+0000 7f0add59a640 1 -- 172.21.15.162:0/3439866804 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad002e070 con 0x7f0ae00b4680 2024-01-27T03:42:28.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.806+0000 7f0add59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:28.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.806+0000 7f0add59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:28.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.806+0000 7f0add59a640 10 monclient: dump: 2024-01-27T03:42:28.807 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:28.807 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.807 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.807 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0add59a640 10 monclient: _finish_auth 0 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0add59a640 10 monclient: _check_auth_tickets 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0add59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0add59a640 1 -- 172.21.15.162:0/3439866804 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad00315a0 con 0x7f0ae00b4680 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 5 monclient: authenticate success, global_id 4121 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0add59a640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:28.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0add59a640 1 -- 172.21.15.162:0/3439866804 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0031930 con 0x7f0ae00b4680 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0add59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 4 set_mon_vals no callback set 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0add59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0add59a640 10 monclient: dump: 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:28.809 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:28.810 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 10 monclient: shutdown 2024-01-27T03:42:28.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:28.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/3439866804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae00b4680 msgr2=0x7f0ae0142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 1 --2- 172.21.15.162:0/3439866804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae00b4680 0x7f0ae0142970 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00097a0 tx=0x7f0ad002b840 comp rx=0 tx=0).stop 2024-01-27T03:42:28.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/3439866804 shutdown_connections 2024-01-27T03:42:28.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 1 --2- 172.21.15.162:0/3439866804 >> v2:172.21.15.162:3300/0 conn(0x7f0ae00b4a60 0x7f0ae00b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 1 --2- 172.21.15.162:0/3439866804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae00b4680 0x7f0ae0142970 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.813 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/3439866804 >> 172.21.15.162:0/3439866804 conn(0x7f0ae0066490 msgr2=0x7f0ae00668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/3439866804 shutdown_connections 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.807+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/3439866804 wait complete. 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.808+0000 7f0ae4f3e640 10 monclient: build_initial_monmap 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.808+0000 7f0ae4f3e640 10 monclient: monmap: 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.808+0000 7f0ae4f3e640 1 librados: starting msgr at 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.808+0000 7f0ae4f3e640 1 librados: starting objecter 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 Processor -- start 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 -- start start 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 librados: setting wanted keys 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 librados: calling monclient init 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient: init 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ae00b4680 0x7f0ae013dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae00b4680 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 0x7f0ae0137e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae0142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae00ab3d0 0x7f0ae0138370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae00ab3d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae00abea0 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae00acef0 con 0x7f0ae00b4680 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae00ab980 con 0x7f0ae0142eb0 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.809+0000 7f0ae4f3e640 10 monclient(hunting): authenticate will time out at 1441.153475s 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0aded9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae00ab3d0 0x7f0ae0138370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0ade59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ae00b4680 0x7f0ae013dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0addd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 0x7f0ae0137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0addd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 0x7f0ae0137e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.162:49686/0 (socket says 172.21.15.162:49686) 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0addd9b640 1 -- 172.21.15.162:0/404512893 learned_addr learned my addr 172.21.15.162:0/404512893 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0addd9b640 10 monclient(hunting): get_auth_request con 0x7f0ae0142eb0 auth_method 0 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0addd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:28.816 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0addd9b640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0addd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0aded9d640 10 monclient(hunting): get_auth_request con 0x7f0ae00ab3d0 auth_method 0 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0aded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0aded9d640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0aded9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0ade59c640 10 monclient(hunting): get_auth_request con 0x7f0ae00b4680 auth_method 0 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0ade59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0ade59c640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.810+0000 7f0ade59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0aded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0aded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0aded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0addd9b640 1 -- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 msgr2=0x7f0ae0137e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0addd9b640 1 -- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 msgr2=0x7f0ae0137e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-27T03:42:28.817 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0addd9b640 1 --2- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 0x7f0ae0137e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-27T03:42:28.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0addd9b640 1 --2- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 0x7f0ae0137e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T03:42:28.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0ade59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:28.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0ade59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:28.818 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f0ade59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:28.818 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:28.811+0000 7f8733841040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T03:42:28.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 10 monclient(hunting): handle_auth_done global_id 4131 payload 824 2024-01-27T03:42:28.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 1 -- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 msgr2=0x7f0ae0137e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T03:42:28.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 1 --2- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 0x7f0ae0137e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 1 -- 172.21.15.162:0/404512893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ae00b4680 msgr2=0x7f0ae013dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:28.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 1 --2- 172.21.15.162:0/404512893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ae00b4680 0x7f0ae013dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:28.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 10 monclient: _finish_hunting 0 2024-01-27T03:42:28.820 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 1 monclient: found mon.noname-a 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.813+0000 7f0aded9d640 1 -- 172.21.15.162:0/404512893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad0009400 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.815+0000 7f0aded9d640 1 --2- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae00ab3d0 0x7f0ae0138370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0143300 tx=0x7f0ad800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.815+0000 7f0aceffd640 10 client.?.objecter ms_handle_connect 0x7f0ae00ab3d0 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.815+0000 7f0aceffd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 1 -- 172.21.15.162:0/404512893 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8003b70 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 10 monclient: dump: 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:28.821 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 10 monclient: _finish_auth 0 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 10 monclient: _check_auth_tickets 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 1 -- 172.21.15.162:0/404512893 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad8003cd0 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 1 -- 172.21.15.162:0/404512893 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8005310 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0ae4f3e640 5 monclient: authenticate success, global_id 4131 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0ae4f3e640 10 monclient: _renew_subs 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0ae4f3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/404512893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae00aa820 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.822 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0acffff640 4 set_mon_vals no callback set 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0aceffd640 10 monclient: dump: 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0ae4f3e640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:28.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0ae4f3e640 10 monclient: _renew_subs 2024-01-27T03:42:28.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0ae4f3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/404512893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae01d7200 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.819+0000 7f0ae4f3e640 1 librados: init done 2024-01-27T03:42:28.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.820+0000 7f0ae4f3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:28.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.820+0000 7f0ae4f3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:28.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.820+0000 7f0ae4f3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.820+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/404512893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae00acd30 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.911+0000 7f0aceffd640 1 -- 172.21.15.162:0/404512893 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0ad8005470 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.915+0000 7f0aceffd640 1 -- 172.21.15.162:0/404512893 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f0ad80258c0 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.915+0000 7f0aceffd640 10 client.4131.objecter ms_dispatch 0x7f0ae0000b90 osd_map(5..5 src has 1..5) v4 2024-01-27T03:42:28.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.915+0000 7f0aceffd640 3 client.4131.objecter handle_osd_map got epochs [5,5] > 0 2024-01-27T03:42:28.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.915+0000 7f0aceffd640 3 client.4131.objecter handle_osd_map decoding full epoch 5 2024-01-27T03:42:28.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.915+0000 7f0aceffd640 20 client.4131.objecter dump_active .. 0 homeless 2024-01-27T03:42:28.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.915+0000 7f0aceffd640 1 -- 172.21.15.162:0/404512893 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7f0ad8003e40 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.915+0000 7f0aceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:28.916 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.915+0000 7f0aceffd640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:28.955 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:28.955+0000 7f8733841040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T03:42:28.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.988+0000 7f0ae4f3e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2b5190b6-abce-4528-9ea6-8363bdd89328", "id": 3}] 2024-01-27T03:42:28.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.988+0000 7f0ae4f3e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2b5190b6-abce-4528-9ea6-8363bdd89328", "id": 3}] 2024-01-27T03:42:28.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.988+0000 7f0ae4f3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:28.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:28.988+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/404512893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2b5190b6-abce-4528-9ea6-8363bdd89328", "id": 3} v 0) v1 -- 0x7f0ae004ea50 con 0x7f0ae00ab3d0 2024-01-27T03:42:28.998 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:28.998+0000 7f8733841040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T03:42:29.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.016+0000 7f0aceffd640 1 -- 172.21.15.162:0/404512893 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2b5190b6-abce-4528-9ea6-8363bdd89328", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0ad8025430 con 0x7f0ae00ab3d0 2024-01-27T03:42:29.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.016+0000 7f0aceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2b5190b6-abce-4528-9ea6-8363bdd89328", "id": 3}] 2024-01-27T03:42:29.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.016+0000 7f0aceffd640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:29.017 INFO:teuthology.orchestra.run.smithi162.stdout:3 2024-01-27T03:42:29.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 10 librados: watch_flush enter 2024-01-27T03:42:29.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 10 librados: watch_flush exit 2024-01-27T03:42:29.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 20 client.4131.objecter shutdown clearing up homeless session... 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 10 client.4131.objecter successfully canceled tick 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 10 monclient: shutdown 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae00ab3d0 msgr2=0x7f0ae0138370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 --2- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae00ab3d0 0x7f0ae0138370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0143300 tx=0x7f0ad800cb90 comp rx=0 tx=0).stop 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/404512893 shutdown_connections 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 --2- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ae0142eb0 0x7f0ae0137e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 --2- 172.21.15.162:0/404512893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae00ab3d0 0x7f0ae0138370 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 --2- 172.21.15.162:0/404512893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ae00b4680 0x7f0ae013dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/404512893 >> 172.21.15.162:0/404512893 conn(0x7f0ae0066490 msgr2=0x7f0ae00b57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/404512893 shutdown_connections 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 -- 172.21.15.162:0/404512893 wait complete. 2024-01-27T03:42:29.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.017+0000 7f0ae4f3e640 1 librados: shutdown 2024-01-27T03:42:29.029 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new ea281cce-cc93-4030-adbe-ee9e16296537 4 2024-01-27T03:42:29.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:29.048+0000 7f1407c0f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-27T03:42:29.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:29.099+0000 7f1407c0f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-27T03:42:29.149 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:29.148+0000 7f8733841040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.171+0000 7ffa81ce7640 10 monclient: get_monmap_and_config 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.171+0000 7ffa81ce7640 10 monclient: build_initial_monmap 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.171+0000 7ffa81ce7640 10 monclient: monmap: 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:42:29.172304+0000 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:42:29.172304+0000 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:29.172 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:29.173 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:29.173 INFO:teuthology.orchestra.run.smithi162.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:29.173 INFO:teuthology.orchestra.run.smithi162.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:29.173 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.172+0000 7ffa81ce7640 1 Processor -- start 2024-01-27T03:42:29.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.172+0000 7ffa81ce7640 1 -- start start 2024-01-27T03:42:29.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.172+0000 7ffa81ce7640 10 monclient: init 2024-01-27T03:42:29.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:29.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-01-27T03:42:29.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa7c0b4680 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:29.175 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ffa7c0b4a60 0x7ffa7c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 10 monclient(hunting): picked mon.noname-c con 0x7ffa7c0b4a60 addr v2:172.21.15.162:3300/0 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b3b30 0x7ffa7c0b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 10 monclient(hunting): picked mon.noname-b con 0x7ffa7c0b3b30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7c0afbe0 con 0x7ffa7c0b4680 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa7c142240 con 0x7ffa7c0b4a60 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c0b0890 con 0x7ffa7c0b3b30 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa81ce7640 10 monclient(hunting): authenticate will time out at 1441.517080s 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa7bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b3b30 0x7ffa7c0b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa7affd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ffa7c0b4a60 0x7ffa7c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa7b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ffa7c0b4680 0x7ffa7c142970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.162:46562/0 (socket says 172.21.15.162:46562) 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.173+0000 7ffa7b7fe640 1 -- 172.21.15.162:0/1044690147 learned_addr learned my addr 172.21.15.162:0/1044690147 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-01-27T03:42:29.176 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7affd640 10 monclient(hunting): get_auth_request con 0x7ffa7c0b4a60 auth_method 0 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7affd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7bfff640 10 monclient(hunting): get_auth_request con 0x7ffa7c0b3b30 auth_method 0 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.174+0000 7ffa7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.176+0000 7ffa7affd640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-01-27T03:42:29.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.176+0000 7ffa7affd640 1 -- 172.21.15.162:0/1044690147 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b3b30 msgr2=0x7ffa7c0b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:29.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.176+0000 7ffa7affd640 1 --2- 172.21.15.162:0/1044690147 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b3b30 0x7ffa7c0b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.176+0000 7ffa7affd640 1 -- 172.21.15.162:0/1044690147 >> v1:172.21.15.110:6789/0 conn(0x7ffa7c0b4680 legacy=0x7ffa7c142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.176+0000 7ffa7affd640 10 monclient: _finish_hunting 0 2024-01-27T03:42:29.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.176+0000 7ffa7affd640 1 monclient: found mon.noname-c 2024-01-27T03:42:29.178 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.176+0000 7ffa7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:29.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.176+0000 7ffa7affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:42:29.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.176+0000 7ffa7affd640 1 -- 172.21.15.162:0/1044690147 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c0b0b00 con 0x7ffa7c0b4a60 2024-01-27T03:42:29.179 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.178+0000 7ffa7affd640 1 --2- 172.21.15.162:0/1044690147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b4a60 0x7ffa7c0b1470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0097a0 tx=0x7ffa6c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b67889766d466a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 1 -- 172.21.15.162:0/1044690147 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c02e070 con 0x7ffa7c0b4a60 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 10 monclient: dump: 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.182 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 10 monclient: _finish_auth 0 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 1 -- 172.21.15.162:0/1044690147 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6c0315a0 con 0x7ffa7c0b4a60 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa81ce7640 5 monclient: authenticate success, global_id 4117 2024-01-27T03:42:29.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa81ce7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa81ce7640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 1 -- 172.21.15.162:0/1044690147 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c031930 con 0x7ffa7c0b4a60 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa81ce7640 4 set_mon_vals no callback set 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.181+0000 7ffa7a7fc640 10 monclient: dump: 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.184 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.185 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.185 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.185 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.185 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.185 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.185 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 10 monclient: shutdown 2024-01-27T03:42:29.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:29.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 1 -- 172.21.15.162:0/1044690147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b4a60 msgr2=0x7ffa7c0b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 1 --2- 172.21.15.162:0/1044690147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b4a60 0x7ffa7c0b1470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0097a0 tx=0x7ffa6c02b840 comp rx=0 tx=0).stop 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 1 -- 172.21.15.162:0/1044690147 shutdown_connections 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 1 --2- 172.21.15.162:0/1044690147 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b3b30 0x7ffa7c0b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 1 --2- 172.21.15.162:0/1044690147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b4a60 0x7ffa7c0b1470 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 1 -- 172.21.15.162:0/1044690147 >> 172.21.15.162:0/1044690147 conn(0x7ffa7c066490 msgr2=0x7ffa7c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 1 -- 172.21.15.162:0/1044690147 shutdown_connections 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.182+0000 7ffa81ce7640 1 -- 172.21.15.162:0/1044690147 wait complete. 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.183+0000 7ffa81ce7640 10 monclient: build_initial_monmap 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.183+0000 7ffa81ce7640 10 monclient: monmap: 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:29.190 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.183+0000 7ffa81ce7640 1 librados: starting msgr at 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.183+0000 7ffa81ce7640 1 librados: starting objecter 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.183+0000 7ffa81ce7640 1 Processor -- start 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 1 -- start start 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 1 librados: setting wanted keys 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 1 librados: calling monclient init 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient: init 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b3b30 0x7ffa7c13a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient(hunting): picked mon.noname-b con 0x7ffa7c0b3b30 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b4680 0x7ffa7c0ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient(hunting): picked mon.noname-c con 0x7ffa7c0b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa7c0aa940 0x7ffa7c137ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient(hunting): picked mon.noname-a con 0x7ffa7c0aa940 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c0ab820 con 0x7ffa7c0aa940 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c0aba40 con 0x7ffa7c0b3b30 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c1381e0 con 0x7ffa7c0b4680 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:29.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.184+0000 7ffa81ce7640 10 monclient(hunting): authenticate will time out at 1441.528394s 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b3b30 0x7ffa7c13a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa7c0aa940 0x7ffa7c137ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b4680 0x7ffa7c0ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa7c0aa940 0x7ffa7c137ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.162:54090/0 (socket says 172.21.15.162:54090) 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 1 -- 172.21.15.162:0/3305916036 learned_addr learned my addr 172.21.15.162:0/3305916036 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 10 monclient(hunting): get_auth_request con 0x7ffa7c0aa940 auth_method 0 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7affd640 10 monclient(hunting): get_auth_request con 0x7ffa7c0b4680 auth_method 0 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7affd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7b7fe640 1 -- 172.21.15.162:0/3305916036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b3b30 msgr2=0x7ffa7c13a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7b7fe640 1 -- 172.21.15.162:0/3305916036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b3b30 msgr2=0x7ffa7c13a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7b7fe640 1 --2- 172.21.15.162:0/3305916036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b3b30 0x7ffa7c13a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7b7fe640 1 --2- 172.21.15.162:0/3305916036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b3b30 0x7ffa7c13a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.185+0000 7ffa7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.186+0000 7ffa7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.186+0000 7ffa7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.186+0000 7ffa7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 10 monclient(hunting): handle_auth_done global_id 4137 payload 824 2024-01-27T03:42:29.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 1 -- 172.21.15.162:0/3305916036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b4680 msgr2=0x7ffa7c0ab2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 1 --2- 172.21.15.162:0/3305916036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b4680 0x7ffa7c0ab2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 1 -- 172.21.15.162:0/3305916036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b3b30 msgr2=0x7ffa7c13a840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T03:42:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 1 --2- 172.21.15.162:0/3305916036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b3b30 0x7ffa7c13a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 10 monclient: _finish_hunting 0 2024-01-27T03:42:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 1 monclient: found mon.noname-a 2024-01-27T03:42:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.188+0000 7ffa7bfff640 1 -- 172.21.15.162:0/3305916036 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c009400 con 0x7ffa7c0aa940 2024-01-27T03:42:29.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.191+0000 7ffa7bfff640 1 --2- 172.21.15.162:0/3305916036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa7c0aa940 0x7ffa7c137ca0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400efc0 tx=0x7ffa7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:29.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.191+0000 7ffa63fff640 10 client.?.objecter ms_handle_connect 0x7ffa7c0aa940 2024-01-27T03:42:29.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.191+0000 7ffa63fff640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:29.197 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 1 -- 172.21.15.162:0/3305916036 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74014480 con 0x7ffa7c0aa940 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 10 monclient: dump: 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.198 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 10 monclient: _finish_auth 0 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 10 monclient: _check_auth_tickets 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 1 -- 172.21.15.162:0/3305916036 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7400f040 con 0x7ffa7c0aa940 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa81ce7640 5 monclient: authenticate success, global_id 4137 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa81ce7640 10 monclient: _renew_subs 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa81ce7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa81ce7640 1 -- 172.21.15.162:0/3305916036 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7c138460 con 0x7ffa7c0aa940 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 1 -- 172.21.15.162:0/3305916036 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74014820 con 0x7ffa7c0aa940 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa63fff640 10 monclient: dump: 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa79ffb640 4 set_mon_vals no callback set 2024-01-27T03:42:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa81ce7640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa81ce7640 10 monclient: _renew_subs 2024-01-27T03:42:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa81ce7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa81ce7640 1 -- 172.21.15.162:0/3305916036 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7c1d7170 con 0x7ffa7c0aa940 2024-01-27T03:42:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.195+0000 7ffa81ce7640 1 librados: init done 2024-01-27T03:42:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.196+0000 7ffa81ce7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.196+0000 7ffa81ce7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.196+0000 7ffa81ce7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.196+0000 7ffa81ce7640 1 -- 172.21.15.162:0/3305916036 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7c074d10 con 0x7ffa7c0aa940 2024-01-27T03:42:29.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.200+0000 7ffa63fff640 1 -- 172.21.15.162:0/3305916036 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ffa74014d10 con 0x7ffa7c0aa940 2024-01-27T03:42:29.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.200+0000 7ffa63fff640 1 -- 172.21.15.162:0/3305916036 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7ffa740138f0 con 0x7ffa7c0aa940 2024-01-27T03:42:29.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.200+0000 7ffa63fff640 10 client.4137.objecter ms_dispatch 0x7ffa7c000b90 osd_map(6..6 src has 1..6) v4 2024-01-27T03:42:29.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.200+0000 7ffa63fff640 3 client.4137.objecter handle_osd_map got epochs [6,6] > 0 2024-01-27T03:42:29.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.200+0000 7ffa63fff640 3 client.4137.objecter handle_osd_map decoding full epoch 6 2024-01-27T03:42:29.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.200+0000 7ffa63fff640 20 client.4137.objecter dump_active .. 0 homeless 2024-01-27T03:42:29.218 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:29.217+0000 7f1407c0f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-27T03:42:29.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.325+0000 7ffa63fff640 1 -- 172.21.15.162:0/3305916036 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7ffa7403bdb0 con 0x7ffa7c0aa940 2024-01-27T03:42:29.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.325+0000 7ffa63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:29.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.325+0000 7ffa63fff640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:29.332 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:29.331+0000 7f1407c0f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-27T03:42:29.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.396+0000 7ffa81ce7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ea281cce-cc93-4030-adbe-ee9e16296537", "id": 4}] 2024-01-27T03:42:29.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.396+0000 7ffa81ce7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ea281cce-cc93-4030-adbe-ee9e16296537", "id": 4}] 2024-01-27T03:42:29.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.396+0000 7ffa81ce7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.396+0000 7ffa81ce7640 1 -- 172.21.15.162:0/3305916036 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ea281cce-cc93-4030-adbe-ee9e16296537", "id": 4} v 0) v1 -- 0x7ffa7c04ea50 con 0x7ffa7c0aa940 2024-01-27T03:42:29.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.428+0000 7ffa63fff640 1 -- 172.21.15.162:0/3305916036 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea281cce-cc93-4030-adbe-ee9e16296537", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffa7402a030 con 0x7ffa7c0aa940 2024-01-27T03:42:29.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.428+0000 7ffa63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ea281cce-cc93-4030-adbe-ee9e16296537", "id": 4}] 2024-01-27T03:42:29.429 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.428+0000 7ffa63fff640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:29.429 INFO:teuthology.orchestra.run.smithi162.stdout:4 2024-01-27T03:42:29.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.428+0000 7ffa81ce7640 10 librados: watch_flush enter 2024-01-27T03:42:29.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 10 librados: watch_flush exit 2024-01-27T03:42:29.430 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 20 client.4137.objecter shutdown clearing up homeless session... 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 10 client.4137.objecter successfully canceled tick 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 10 monclient: shutdown 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 -- 172.21.15.162:0/3305916036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa7c0aa940 msgr2=0x7ffa7c137ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 --2- 172.21.15.162:0/3305916036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa7c0aa940 0x7ffa7c137ca0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400efc0 tx=0x7ffa7400c490 comp rx=0 tx=0).stop 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 -- 172.21.15.162:0/3305916036 shutdown_connections 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 --2- 172.21.15.162:0/3305916036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa7c0b4680 0x7ffa7c0ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 --2- 172.21.15.162:0/3305916036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa7c0aa940 0x7ffa7c137ca0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 --2- 172.21.15.162:0/3305916036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa7c0b3b30 0x7ffa7c13a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 -- 172.21.15.162:0/3305916036 >> 172.21.15.162:0/3305916036 conn(0x7ffa7c066490 msgr2=0x7ffa7c0b73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 -- 172.21.15.162:0/3305916036 shutdown_connections 2024-01-27T03:42:29.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 -- 172.21.15.162:0/3305916036 wait complete. 2024-01-27T03:42:29.432 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.429+0000 7ffa81ce7640 1 librados: shutdown 2024-01-27T03:42:29.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47 5 2024-01-27T03:42:29.475 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:29.474+0000 7f1407c0f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-27T03:42:29.516 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:29.515+0000 7f1407c0f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-27T03:42:29.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.538+0000 7f1c2e0a9640 10 monclient: get_monmap_and_config 2024-01-27T03:42:29.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.538+0000 7f1c2e0a9640 10 monclient: build_initial_monmap 2024-01-27T03:42:29.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.539+0000 7f1c2e0a9640 10 monclient: monmap: 2024-01-27T03:42:29.539 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:29.539 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.539 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:42:29.539460+0000 2024-01-27T03:42:29.539 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:42:29.539460+0000 2024-01-27T03:42:29.539 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:29.540 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.540 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:29.540 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:29.540 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:29.540 INFO:teuthology.orchestra.run.smithi162.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:29.540 INFO:teuthology.orchestra.run.smithi162.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:29.540 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.539+0000 7f1c2e0a9640 1 Processor -- start 2024-01-27T03:42:29.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.539+0000 7f1c2e0a9640 1 -- start start 2024-01-27T03:42:29.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.539+0000 7f1c2e0a9640 10 monclient: init 2024-01-27T03:42:29.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:29.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-01-27T03:42:29.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1c280b4680 0x7f1c28142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 10 monclient(hunting): picked mon.noname-c con 0x7f1c280b4680 addr v2:172.21.15.162:3300/0 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c280b4a60 0x7f1c280b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 10 monclient(hunting): picked mon.noname-a con 0x7f1c280b4a60 addr v2:172.21.15.110:3300/0 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c28142eb0 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c280afbe0 con 0x7f1c28142eb0 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c28142240 con 0x7f1c280b4a60 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c280b0890 con 0x7f1c280b4680 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.540+0000 7f1c2e0a9640 10 monclient(hunting): authenticate will time out at 1441.884338s 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c277fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1c280b4680 0x7f1c28142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c1effd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c280b4a60 0x7f1c280b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c27fff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f1c28142eb0 0x7f1c280b19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52992/0 (socket says 172.21.15.162:52992) 2024-01-27T03:42:29.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c27fff640 1 -- 172.21.15.162:0/4255299561 learned_addr learned my addr 172.21.15.162:0/4255299561 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c1effd640 10 monclient(hunting): get_auth_request con 0x7f1c280b4a60 auth_method 0 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c1effd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c1effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c277fe640 10 monclient(hunting): get_auth_request con 0x7f1c280b4680 auth_method 0 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c277fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.541+0000 7f1c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.542+0000 7f1c1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.542+0000 7f1c1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.542+0000 7f1c1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.542+0000 7f1c277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.542+0000 7f1c277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.542+0000 7f1c277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.543+0000 7f1c1effd640 10 monclient(hunting): handle_auth_done global_id 4140 payload 274 2024-01-27T03:42:29.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.543+0000 7f1c1effd640 1 -- 172.21.15.162:0/4255299561 >> v2:172.21.15.162:3300/0 conn(0x7f1c280b4680 msgr2=0x7f1c28142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:29.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.543+0000 7f1c1effd640 1 --2- 172.21.15.162:0/4255299561 >> v2:172.21.15.162:3300/0 conn(0x7f1c280b4680 0x7f1c28142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.543+0000 7f1c1effd640 1 -- 172.21.15.162:0/4255299561 >> v1:172.21.15.162:6789/0 conn(0x7f1c28142eb0 legacy=0x7f1c280b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.543+0000 7f1c1effd640 10 monclient: _finish_hunting 0 2024-01-27T03:42:29.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.543+0000 7f1c1effd640 1 monclient: found mon.noname-a 2024-01-27T03:42:29.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.543+0000 7f1c1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:29.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.543+0000 7f1c1effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.543+0000 7f1c1effd640 1 -- 172.21.15.162:0/4255299561 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280b0b00 con 0x7f1c280b4a60 2024-01-27T03:42:29.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.545+0000 7f1c1effd640 1 --2- 172.21.15.162:0/4255299561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c280b4a60 0x7f1c280b1470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c280b4ee0 tx=0x7f1c1002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11c363c21ec2acc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:29.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.548+0000 7f1c26ffd640 1 -- 172.21.15.162:0/4255299561 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1002e070 con 0x7f1c280b4a60 2024-01-27T03:42:29.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.548+0000 7f1c26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:29.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 10 monclient: dump: 2024-01-27T03:42:29.549 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 10 monclient: _finish_auth 0 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 10 monclient: _check_auth_tickets 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:29.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 1 -- 172.21.15.162:0/4255299561 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c100315a0 con 0x7f1c280b4a60 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 1 -- 172.21.15.162:0/4255299561 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10031930 con 0x7f1c280b4a60 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 5 monclient: authenticate success, global_id 4140 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 4 set_mon_vals no callback set 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c26ffd640 10 monclient: dump: 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.551 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.552 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.552 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.552 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.552 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.552 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.552 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 10 monclient: shutdown 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/4255299561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c280b4a60 msgr2=0x7f1c280b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 1 --2- 172.21.15.162:0/4255299561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c280b4a60 0x7f1c280b1470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c280b4ee0 tx=0x7f1c1002b840 comp rx=0 tx=0).stop 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/4255299561 shutdown_connections 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 1 --2- 172.21.15.162:0/4255299561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c280b4a60 0x7f1c280b1470 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 1 --2- 172.21.15.162:0/4255299561 >> v2:172.21.15.162:3300/0 conn(0x7f1c280b4680 0x7f1c28142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/4255299561 >> 172.21.15.162:0/4255299561 conn(0x7f1c28066490 msgr2=0x7f1c280668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/4255299561 shutdown_connections 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.549+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/4255299561 wait complete. 2024-01-27T03:42:29.559 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.550+0000 7f1c2e0a9640 10 monclient: build_initial_monmap 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.550+0000 7f1c2e0a9640 10 monclient: monmap: 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.550+0000 7f1c2e0a9640 1 librados: starting msgr at 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.550+0000 7f1c2e0a9640 1 librados: starting objecter 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.550+0000 7f1c2e0a9640 1 Processor -- start 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 1 -- start start 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 1 librados: setting wanted keys 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 1 librados: calling monclient init 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient: init 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c280b4680 0x7f1c280ab470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient(hunting): picked mon.noname-b con 0x7f1c280b4680 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c28142eb0 0x7f1c2813e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.561 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient(hunting): picked mon.noname-a con 0x7f1c28142eb0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c280aaf90 0x7f1c281d72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient(hunting): picked mon.noname-c con 0x7f1c280aaf90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280abae0 con 0x7f1c28142eb0 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280ad010 con 0x7f1c280b4680 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280abf70 con 0x7f1c280aaf90 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.551+0000 7f1c2e0a9640 10 monclient(hunting): authenticate will time out at 1441.895474s 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c28142eb0 0x7f1c2813e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.562 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c277fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c280b4680 0x7f1c280ab470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c27fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c280aaf90 0x7f1c281d72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c28142eb0 0x7f1c2813e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.162:54110/0 (socket says 172.21.15.162:54110) 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c1effd640 1 -- 172.21.15.162:0/3203419865 learned_addr learned my addr 172.21.15.162:0/3203419865 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c1effd640 10 monclient(hunting): get_auth_request con 0x7f1c28142eb0 auth_method 0 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c1effd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c1effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c277fe640 10 monclient(hunting): get_auth_request con 0x7f1c280b4680 auth_method 0 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c277fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c27fff640 10 monclient(hunting): get_auth_request con 0x7f1c280aaf90 auth_method 0 2024-01-27T03:42:29.563 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c27fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.552+0000 7f1c27fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.553+0000 7f1c1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.553+0000 7f1c1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.553+0000 7f1c1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.553+0000 7f1c27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.553+0000 7f1c27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.553+0000 7f1c27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.553+0000 7f1c277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.553+0000 7f1c277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.553+0000 7f1c277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 10 monclient(hunting): handle_auth_done global_id 4143 payload 824 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 1 -- 172.21.15.162:0/3203419865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c280aaf90 msgr2=0x7f1c281d72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.564 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 1 --2- 172.21.15.162:0/3203419865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c280aaf90 0x7f1c281d72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 1 -- 172.21.15.162:0/3203419865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c280b4680 msgr2=0x7f1c280ab470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 1 --2- 172.21.15.162:0/3203419865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c280b4680 0x7f1c280ab470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 10 monclient: _finish_hunting 0 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 1 monclient: found mon.noname-a 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.555+0000 7f1c1effd640 1 -- 172.21.15.162:0/3203419865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c10009400 con 0x7f1c28142eb0 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.558+0000 7f1c1effd640 1 --2- 172.21.15.162:0/3203419865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c28142eb0 0x7f1c2813e6e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009890 tx=0x7f1c100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.558+0000 7f1c24ff9640 10 client.?.objecter ms_handle_connect 0x7f1c28142eb0 2024-01-27T03:42:29.565 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.558+0000 7f1c24ff9640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:29.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c24ff9640 1 -- 172.21.15.162:0/3203419865 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10032840 con 0x7f1c28142eb0 2024-01-27T03:42:29.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.566 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c24ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c24ff9640 10 monclient: dump: 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c24ff9640 10 monclient: _finish_auth 0 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c24ff9640 10 monclient: _check_auth_tickets 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c2e0a9640 5 monclient: authenticate success, global_id 4143 2024-01-27T03:42:29.567 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c2e0a9640 10 monclient: _renew_subs 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c2e0a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/3203419865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c281d77e0 con 0x7f1c28142eb0 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c2e0a9640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c2e0a9640 10 monclient: _renew_subs 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c2e0a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/3203419865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c281d7ce0 con 0x7f1c28142eb0 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.560+0000 7f1c2e0a9640 1 librados: init done 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.561+0000 7f1c2e0a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.561+0000 7f1c2e0a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.561+0000 7f1c2e0a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.568 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.561+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/3203419865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c280abc50 con 0x7f1c28142eb0 2024-01-27T03:42:29.657 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 1 -- 172.21.15.162:0/3203419865 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c100329a0 con 0x7f1c28142eb0 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 1 -- 172.21.15.162:0/3203419865 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10032d10 con 0x7f1c28142eb0 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 10 monclient: dump: 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.658 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c267fc640 4 set_mon_vals no callback set 2024-01-27T03:42:29.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 1 -- 172.21.15.162:0/3203419865 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1c100457a0 con 0x7f1c28142eb0 2024-01-27T03:42:29.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 1 -- 172.21.15.162:0/3203419865 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f1c1003b970 con 0x7f1c28142eb0 2024-01-27T03:42:29.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 10 client.4143.objecter ms_dispatch 0x7f1c28000b90 osd_map(7..7 src has 1..7) v4 2024-01-27T03:42:29.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 3 client.4143.objecter handle_osd_map got epochs [7,7] > 0 2024-01-27T03:42:29.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 3 client.4143.objecter handle_osd_map decoding full epoch 7 2024-01-27T03:42:29.659 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.657+0000 7f1c24ff9640 20 client.4143.objecter dump_active .. 0 homeless 2024-01-27T03:42:29.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.661+0000 7f1c24ff9640 1 -- 172.21.15.162:0/3203419865 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7f1c1004e3f0 con 0x7f1c28142eb0 2024-01-27T03:42:29.661 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.661+0000 7f1c24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:29.662 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.661+0000 7f1c24ff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:29.663 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:29.662+0000 7f1407c0f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-27T03:42:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.732+0000 7f1c2e0a9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47", "id": 5}] 2024-01-27T03:42:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.732+0000 7f1c2e0a9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47", "id": 5}] 2024-01-27T03:42:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.732+0000 7f1c2e0a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:29.733 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.732+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/3203419865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47", "id": 5} v 0) v1 -- 0x7f1c2804ea50 con 0x7f1c28142eb0 2024-01-27T03:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.762+0000 7f1c24ff9640 1 -- 172.21.15.162:0/3203419865 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1c1003bba0 con 0x7f1c28142eb0 2024-01-27T03:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.762+0000 7f1c24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47", "id": 5}] 2024-01-27T03:42:29.762 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.762+0000 7f1c24ff9640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:29.762 INFO:teuthology.orchestra.run.smithi162.stdout:5 2024-01-27T03:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 10 librados: watch_flush enter 2024-01-27T03:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 10 librados: watch_flush exit 2024-01-27T03:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 20 client.4143.objecter shutdown clearing up homeless session... 2024-01-27T03:42:29.764 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 10 client.4143.objecter successfully canceled tick 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 10 monclient: shutdown 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/3203419865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c28142eb0 msgr2=0x7f1c2813e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 --2- 172.21.15.162:0/3203419865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c28142eb0 0x7f1c2813e6e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009890 tx=0x7f1c100046c0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/3203419865 shutdown_connections 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 --2- 172.21.15.162:0/3203419865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c280aaf90 0x7f1c281d72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 --2- 172.21.15.162:0/3203419865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c28142eb0 0x7f1c2813e6e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 --2- 172.21.15.162:0/3203419865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c280b4680 0x7f1c280ab470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/3203419865 >> 172.21.15.162:0/3203419865 conn(0x7f1c28066490 msgr2=0x7f1c28066870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/3203419865 shutdown_connections 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 -- 172.21.15.162:0/3203419865 wait complete. 2024-01-27T03:42:29.765 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.763+0000 7f1c2e0a9640 1 librados: shutdown 2024-01-27T03:42:29.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 26b0b3bd-a412-4d75-a5ab-86df55753f8a 6 2024-01-27T03:42:29.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.879+0000 7f68c52d0640 10 monclient: get_monmap_and_config 2024-01-27T03:42:29.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.879+0000 7f68c52d0640 10 monclient: build_initial_monmap 2024-01-27T03:42:29.880 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.879+0000 7f68c52d0640 10 monclient: monmap: 2024-01-27T03:42:29.880 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:29.880 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.880 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:42:29.880396+0000 2024-01-27T03:42:29.880 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:42:29.880396+0000 2024-01-27T03:42:29.880 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:29.881 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.881 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:29.881 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:29.881 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:29.881 INFO:teuthology.orchestra.run.smithi162.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:29.881 INFO:teuthology.orchestra.run.smithi162.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:29.881 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.880+0000 7f68c52d0640 1 Processor -- start 2024-01-27T03:42:29.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.880+0000 7f68c52d0640 1 -- start start 2024-01-27T03:42:29.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.880+0000 7f68c52d0640 10 monclient: init 2024-01-27T03:42:29.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:29.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-01-27T03:42:29.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c006a940 0x7f68c013f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 10 monclient(hunting): picked mon.noname-b con 0x7f68c006a940 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:29.883 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68c00afe50 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:29.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68c00b0290 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:29.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c006b5c0 con 0x7f68c00b0290 2024-01-27T03:42:29.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c00af970 con 0x7f68c00afe50 2024-01-27T03:42:29.884 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c006ad20 con 0x7f68c006a940 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68c52d0640 10 monclient(hunting): authenticate will time out at 1442.225076s 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68be7fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f68c00afe50 0x7f68c01455b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:53004/0 (socket says 172.21.15.162:53004) 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68be7fc640 1 -- 172.21.15.162:0/556008109 learned_addr learned my addr 172.21.15.162:0/556008109 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.881+0000 7f68beffd640 1 --2- 172.21.15.162:0/556008109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c006a940 0x7f68c013f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.882+0000 7f68beffd640 10 monclient(hunting): get_auth_request con 0x7f68c006a940 auth_method 0 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.882+0000 7f68beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.882+0000 7f68beffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.882+0000 7f68beffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.882+0000 7f68beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.882+0000 7f68beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.885 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.882+0000 7f68beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.886 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68bdffb640 1 -- 172.21.15.162:0/556008109 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317677796 0 0) 0x7f68c006b5c0 con 0x7f68c00b0290 2024-01-27T03:42:29.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68bdffb640 10 monclient(hunting): _negotiate 2024-01-27T03:42:29.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68bdffb640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68bdffb640 10 monclient(hunting): my global_id is 4146 2024-01-27T03:42:29.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68bdffb640 1 -- 172.21.15.162:0/556008109 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a8003520 con 0x7f68c00b0290 2024-01-27T03:42:29.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68beffd640 10 monclient(hunting): handle_auth_done global_id 4136 payload 274 2024-01-27T03:42:29.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68beffd640 1 -- 172.21.15.162:0/556008109 >> v1:172.21.15.162:6789/0 conn(0x7f68c00afe50 legacy=0x7f68c01455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68beffd640 1 -- 172.21.15.162:0/556008109 >> v1:172.21.15.110:6789/0 conn(0x7f68c00b0290 legacy=0x7f68c014c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.887 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68beffd640 10 monclient: _finish_hunting 0 2024-01-27T03:42:29.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68beffd640 1 monclient: found mon.noname-b 2024-01-27T03:42:29.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:29.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68beffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:42:29.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.884+0000 7f68beffd640 1 -- 172.21.15.162:0/556008109 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c0145cc0 con 0x7f68c006a940 2024-01-27T03:42:29.888 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.886+0000 7f68beffd640 1 --2- 172.21.15.162:0/556008109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c006a940 0x7f68c013f3d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f68b40097d0 tx=0x7f68b402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4c06e7cf82d51da server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:29.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 1 -- 172.21.15.162:0/556008109 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4031800 con 0x7f68c006a940 2024-01-27T03:42:29.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:29.890 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 10 monclient: dump: 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 10 monclient: _finish_auth 0 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 10 monclient: _check_auth_tickets 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:29.891 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 1 -- 172.21.15.162:0/556008109 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b4031960 con 0x7f68c006a940 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 1 -- 172.21.15.162:0/556008109 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4031cf0 con 0x7f68c006a940 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68c52d0640 5 monclient: authenticate success, global_id 4136 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68c52d0640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68c52d0640 4 set_mon_vals no callback set 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.890+0000 7f68bdffb640 10 monclient: dump: 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.892 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.893 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.893 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.893 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.893 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.891+0000 7f68c52d0640 10 monclient: shutdown 2024-01-27T03:42:29.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.891+0000 7f68c52d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:29.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.891+0000 7f68c52d0640 1 -- 172.21.15.162:0/556008109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c006a940 msgr2=0x7f68c013f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.891+0000 7f68c52d0640 1 --2- 172.21.15.162:0/556008109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c006a940 0x7f68c013f3d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f68b40097d0 tx=0x7f68b402b8a0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.896 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.891+0000 7f68c52d0640 1 -- 172.21.15.162:0/556008109 shutdown_connections 2024-01-27T03:42:29.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.891+0000 7f68c52d0640 1 --2- 172.21.15.162:0/556008109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c006a940 0x7f68c013f3d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.891+0000 7f68c52d0640 1 -- 172.21.15.162:0/556008109 >> 172.21.15.162:0/556008109 conn(0x7f68c00b1400 msgr2=0x7f68c00b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:29.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.891+0000 7f68c52d0640 1 -- 172.21.15.162:0/556008109 shutdown_connections 2024-01-27T03:42:29.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.891+0000 7f68c52d0640 1 -- 172.21.15.162:0/556008109 wait complete. 2024-01-27T03:42:29.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.892+0000 7f68c52d0640 10 monclient: build_initial_monmap 2024-01-27T03:42:29.897 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.892+0000 7f68c52d0640 10 monclient: monmap: 2024-01-27T03:42:29.897 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.892+0000 7f68c52d0640 1 librados: starting msgr at 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.892+0000 7f68c52d0640 1 librados: starting objecter 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 Processor -- start 2024-01-27T03:42:29.898 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 -- start start 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 librados: setting wanted keys 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 librados: calling monclient init 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient: init 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68c006a940 0x7f68c0138730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient(hunting): picked mon.noname-b con 0x7f68c006a940 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c00afe50 0x7f68c01dade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient(hunting): picked mon.noname-c con 0x7f68c00afe50 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68c00b0290 0x7f68c0140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient(hunting): picked mon.noname-a con 0x7f68c00b0290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c0070510 con 0x7f68c00b0290 2024-01-27T03:42:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c0071530 con 0x7f68c006a940 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c00717a0 con 0x7f68c00afe50 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.893+0000 7f68c52d0640 10 monclient(hunting): authenticate will time out at 1442.237582s 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68beffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68c006a940 0x7f68c0138730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68be7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c00afe50 0x7f68c01dade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68bf7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68c00b0290 0x7f68c0140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68beffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68c006a940 0x7f68c0138730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:49508/0 (socket says 172.21.15.162:49508) 2024-01-27T03:42:29.900 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68beffd640 1 -- 172.21.15.162:0/3314744650 learned_addr learned my addr 172.21.15.162:0/3314744650 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68beffd640 10 monclient(hunting): get_auth_request con 0x7f68c006a940 auth_method 0 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68beffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68beffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68bf7fe640 10 monclient(hunting): get_auth_request con 0x7f68c00b0290 auth_method 0 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68bf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68be7fc640 10 monclient(hunting): get_auth_request con 0x7f68c00afe50 auth_method 0 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68be7fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.894+0000 7f68be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.895+0000 7f68beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.895+0000 7f68beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.901 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.895+0000 7f68beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.895+0000 7f68be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.895+0000 7f68be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.895+0000 7f68be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.895+0000 7f68bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:29.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.895+0000 7f68bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:29.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.895+0000 7f68bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 10 monclient(hunting): handle_auth_done global_id 4139 payload 824 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 1 -- 172.21.15.162:0/3314744650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68c006a940 msgr2=0x7f68c0138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 1 --2- 172.21.15.162:0/3314744650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68c006a940 0x7f68c0138730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 1 -- 172.21.15.162:0/3314744650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68c00b0290 msgr2=0x7f68c0140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 1 --2- 172.21.15.162:0/3314744650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68c00b0290 0x7f68c0140f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 10 monclient: _finish_hunting 0 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 1 monclient: found mon.noname-c 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68be7fc640 1 -- 172.21.15.162:0/3314744650 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b4009400 con 0x7f68c00afe50 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.897+0000 7f68beffd640 1 --2- 172.21.15.162:0/3314744650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68c006a940 0x7f68c0138730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T03:42:29.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.900+0000 7f68be7fc640 1 --2- 172.21.15.162:0/3314744650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c00afe50 0x7f68c01dade0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f68b000c370 tx=0x7f68b000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.900+0000 7f68a77fe640 10 client.?.objecter ms_handle_connect 0x7f68c00afe50 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.900+0000 7f68a77fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 1 -- 172.21.15.162:0/3314744650 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0005ce0 con 0x7f68c00afe50 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 10 monclient: dump: 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.906 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 10 monclient: _finish_auth 0 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 10 monclient: _check_auth_tickets 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 1 -- 172.21.15.162:0/3314744650 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b0005e40 con 0x7f68c00afe50 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68c52d0640 5 monclient: authenticate success, global_id 4139 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 1 -- 172.21.15.162:0/3314744650 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0007310 con 0x7f68c00afe50 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:29.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68a77fe640 10 monclient: dump: 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:29.908 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68c52d0640 10 monclient: _renew_subs 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68c52d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68c52d0640 1 -- 172.21.15.162:0/3314744650 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c01db440 con 0x7f68c00afe50 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68bd7fa640 4 set_mon_vals no callback set 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68c52d0640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68c52d0640 10 monclient: _renew_subs 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68c52d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68c52d0640 1 -- 172.21.15.162:0/3314744650 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c01db900 con 0x7f68c00afe50 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.903+0000 7f68c52d0640 1 librados: init done 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.904+0000 7f68c52d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.904+0000 7f68c52d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.904+0000 7f68c52d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:29.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.904+0000 7f68c52d0640 1 -- 172.21.15.162:0/3314744650 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c00706c0 con 0x7f68c00afe50 2024-01-27T03:42:29.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.906+0000 7f68a77fe640 1 -- 172.21.15.162:0/3314744650 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f68b0007470 con 0x7f68c00afe50 2024-01-27T03:42:29.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.906+0000 7f68a77fe640 1 -- 172.21.15.162:0/3314744650 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f68b001b9a0 con 0x7f68c00afe50 2024-01-27T03:42:29.910 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.906+0000 7f68a77fe640 10 client.4139.objecter ms_dispatch 0x7f68c0000b90 osd_map(8..8 src has 1..8) v4 2024-01-27T03:42:29.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.906+0000 7f68a77fe640 3 client.4139.objecter handle_osd_map got epochs [8,8] > 0 2024-01-27T03:42:29.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.906+0000 7f68a77fe640 3 client.4139.objecter handle_osd_map decoding full epoch 8 2024-01-27T03:42:29.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:29.906+0000 7f68a77fe640 20 client.4139.objecter dump_active .. 0 homeless 2024-01-27T03:42:29.996 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:29.996+0000 7f8733841040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T03:42:30.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.004+0000 7f68a77fe640 1 -- 172.21.15.162:0/3314744650 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7f68c00706c0 con 0x7f68c00afe50 2024-01-27T03:42:30.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.004+0000 7f68a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:30.004 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.004+0000 7f68a77fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.076+0000 7f68c52d0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "26b0b3bd-a412-4d75-a5ab-86df55753f8a", "id": 6}] 2024-01-27T03:42:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.076+0000 7f68c52d0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "26b0b3bd-a412-4d75-a5ab-86df55753f8a", "id": 6}] 2024-01-27T03:42:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.076+0000 7f68c52d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.076+0000 7f68c52d0640 1 -- 172.21.15.162:0/3314744650 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd new", "uuid": "26b0b3bd-a412-4d75-a5ab-86df55753f8a", "id": 6} v 0) v1 -- 0x7f68c004ea50 con 0x7f68c00afe50 2024-01-27T03:42:30.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.184+0000 7f68a77fe640 1 -- 172.21.15.162:0/3314744650 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26b0b3bd-a412-4d75-a5ab-86df55753f8a", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f68b0017070 con 0x7f68c00afe50 2024-01-27T03:42:30.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.184+0000 7f68a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "26b0b3bd-a412-4d75-a5ab-86df55753f8a", "id": 6}] 2024-01-27T03:42:30.185 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.184+0000 7f68a77fe640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:30.185 INFO:teuthology.orchestra.run.smithi162.stdout:6 2024-01-27T03:42:30.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 10 librados: watch_flush enter 2024-01-27T03:42:30.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 10 librados: watch_flush exit 2024-01-27T03:42:30.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 20 client.4139.objecter shutdown clearing up homeless session... 2024-01-27T03:42:30.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 10 client.4139.objecter successfully canceled tick 2024-01-27T03:42:30.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 10 monclient: shutdown 2024-01-27T03:42:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 -- 172.21.15.162:0/3314744650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c00afe50 msgr2=0x7f68c01dade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 --2- 172.21.15.162:0/3314744650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c00afe50 0x7f68c01dade0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f68b000c370 tx=0x7f68b000c830 comp rx=0 tx=0).stop 2024-01-27T03:42:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 -- 172.21.15.162:0/3314744650 shutdown_connections 2024-01-27T03:42:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 --2- 172.21.15.162:0/3314744650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68c00afe50 0x7f68c01dade0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 --2- 172.21.15.162:0/3314744650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68c00b0290 0x7f68c0140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 --2- 172.21.15.162:0/3314744650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68c006a940 0x7f68c0138730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 -- 172.21.15.162:0/3314744650 >> 172.21.15.162:0/3314744650 conn(0x7f68c00b1400 msgr2=0x7f68c00ae010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:30.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 -- 172.21.15.162:0/3314744650 shutdown_connections 2024-01-27T03:42:30.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 -- 172.21.15.162:0/3314744650 wait complete. 2024-01-27T03:42:30.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.185+0000 7f68c52d0640 1 librados: shutdown 2024-01-27T03:42:30.196 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 7b8b531d-33ed-4bd5-8b97-d5ff4b210c17 7 2024-01-27T03:42:30.278 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:30.277+0000 7f8733841040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T03:42:30.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.295+0000 7feb6907c640 10 monclient: get_monmap_and_config 2024-01-27T03:42:30.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.295+0000 7feb6907c640 10 monclient: build_initial_monmap 2024-01-27T03:42:30.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.295+0000 7feb6907c640 10 monclient: monmap: 2024-01-27T03:42:30.295 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:30.295 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:30.295 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:42:30.295550+0000 2024-01-27T03:42:30.295 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:42:30.295550+0000 2024-01-27T03:42:30.296 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:30.296 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:30.296 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:30.296 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:30.296 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:30.296 INFO:teuthology.orchestra.run.smithi162.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:30.296 INFO:teuthology.orchestra.run.smithi162.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:30.296 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:30.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.295+0000 7feb6907c640 1 Processor -- start 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.295+0000 7feb6907c640 1 -- start start 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.295+0000 7feb6907c640 10 monclient: init 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb640b4680 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feb640b4a60 0x7feb640b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 10 monclient(hunting): picked mon.noname-a con 0x7feb640b4a60 addr v2:172.21.15.110:3300/0 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b3b30 0x7feb640b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 10 monclient(hunting): picked mon.noname-b con 0x7feb640b3b30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb640afbe0 con 0x7feb640b4680 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb64142240 con 0x7feb640b4a60 2024-01-27T03:42:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb640b0890 con 0x7feb640b3b30 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb6907c640 10 monclient(hunting): authenticate will time out at 1442.640178s 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb62d9d640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7feb640b4680 0x7feb64142970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:53012/0 (socket says 172.21.15.162:53012) 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.296+0000 7feb62d9d640 1 -- 172.21.15.162:0/3239248646 learned_addr learned my addr 172.21.15.162:0/3239248646 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6359e640 1 --2- 172.21.15.162:0/3239248646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b3b30 0x7feb640b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6259c640 1 --2- 172.21.15.162:0/3239248646 >> v2:172.21.15.110:3300/0 conn(0x7feb640b4a60 0x7feb640b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6259c640 10 monclient(hunting): get_auth_request con 0x7feb640b4a60 auth_method 0 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6259c640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6359e640 10 monclient(hunting): get_auth_request con 0x7feb640b3b30 auth_method 0 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:30.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6359e640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.297+0000 7feb6359e640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.298+0000 7feb6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.298+0000 7feb6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.298+0000 7feb6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.298+0000 7feb6359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.298+0000 7feb6359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:30.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.298+0000 7feb6359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.299+0000 7feb61d9b640 1 -- 172.21.15.162:0/3239248646 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634074013 0 0) 0x7feb640afbe0 con 0x7feb640b4680 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.299+0000 7feb61d9b640 10 monclient(hunting): _negotiate 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.299+0000 7feb61d9b640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.299+0000 7feb61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.299+0000 7feb61d9b640 10 monclient(hunting): my global_id is 4135 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.299+0000 7feb61d9b640 1 -- 172.21.15.162:0/3239248646 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb4c003520 con 0x7feb640b4680 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.300+0000 7feb6259c640 10 monclient(hunting): handle_auth_done global_id 4152 payload 274 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.300+0000 7feb6259c640 1 -- 172.21.15.162:0/3239248646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b3b30 msgr2=0x7feb640b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.300+0000 7feb6259c640 1 --2- 172.21.15.162:0/3239248646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b3b30 0x7feb640b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.300+0000 7feb6259c640 1 -- 172.21.15.162:0/3239248646 >> v1:172.21.15.162:6789/0 conn(0x7feb640b4680 legacy=0x7feb64142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.300+0000 7feb6259c640 10 monclient: _finish_hunting 0 2024-01-27T03:42:30.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.300+0000 7feb6259c640 1 monclient: found mon.noname-a 2024-01-27T03:42:30.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.300+0000 7feb6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:30.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.300+0000 7feb6259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:30.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.300+0000 7feb6259c640 1 -- 172.21.15.162:0/3239248646 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb640b0b00 con 0x7feb640b4a60 2024-01-27T03:42:30.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.302+0000 7feb6259c640 1 --2- 172.21.15.162:0/3239248646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b4a60 0x7feb640b1470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feb540097a0 tx=0x7feb5402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48268a8c9453d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.304+0000 7feb61d9b640 1 -- 172.21.15.162:0/3239248646 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5402e070 con 0x7feb640b4a60 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.304+0000 7feb61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.304+0000 7feb61d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.304+0000 7feb61d9b640 10 monclient: dump: 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:30.305 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:30.306 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:30.306 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:30.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.304+0000 7feb61d9b640 10 monclient: _finish_auth 0 2024-01-27T03:42:30.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.304+0000 7feb61d9b640 10 monclient: _check_auth_tickets 2024-01-27T03:42:30.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.304+0000 7feb61d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:30.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.304+0000 7feb6907c640 5 monclient: authenticate success, global_id 4152 2024-01-27T03:42:30.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.304+0000 7feb6907c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:42:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb61d9b640 1 -- 172.21.15.162:0/3239248646 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb540315a0 con 0x7feb640b4a60 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb61d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb61d9b640 1 -- 172.21.15.162:0/3239248646 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb54031910 con 0x7feb640b4a60 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb61d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb61d9b640 10 monclient: dump: 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:30.308 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:30.309 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 4 set_mon_vals no callback set 2024-01-27T03:42:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 10 monclient: shutdown 2024-01-27T03:42:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:30.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 1 -- 172.21.15.162:0/3239248646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b4a60 msgr2=0x7feb640b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 1 --2- 172.21.15.162:0/3239248646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b4a60 0x7feb640b1470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feb540097a0 tx=0x7feb5402b840 comp rx=0 tx=0).stop 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 1 -- 172.21.15.162:0/3239248646 shutdown_connections 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 1 --2- 172.21.15.162:0/3239248646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b3b30 0x7feb640b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 1 --2- 172.21.15.162:0/3239248646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b4a60 0x7feb640b1470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.306+0000 7feb6907c640 1 -- 172.21.15.162:0/3239248646 >> 172.21.15.162:0/3239248646 conn(0x7feb64066490 msgr2=0x7feb640668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.307+0000 7feb6907c640 1 -- 172.21.15.162:0/3239248646 shutdown_connections 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.307+0000 7feb6907c640 1 -- 172.21.15.162:0/3239248646 wait complete. 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.307+0000 7feb6907c640 10 monclient: build_initial_monmap 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.307+0000 7feb6907c640 10 monclient: monmap: 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-01-27T03:42:30.310 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-01-27T03:42:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-01-27T03:42:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:30.311 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:30.311 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:30.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.308+0000 7feb6907c640 1 librados: starting msgr at 2024-01-27T03:42:30.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.308+0000 7feb6907c640 1 librados: starting objecter 2024-01-27T03:42:30.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.308+0000 7feb6907c640 1 Processor -- start 2024-01-27T03:42:30.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.308+0000 7feb6907c640 1 -- start start 2024-01-27T03:42:30.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.308+0000 7feb6907c640 1 librados: setting wanted keys 2024-01-27T03:42:30.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.308+0000 7feb6907c640 1 librados: calling monclient init 2024-01-27T03:42:30.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.308+0000 7feb6907c640 10 monclient: init 2024-01-27T03:42:30.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:30.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T03:42:30.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b3b30 0x7feb6413a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:30.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient(hunting): picked mon.noname-a con 0x7feb640b3b30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:30.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b4680 0x7feb640ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:30.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient(hunting): picked mon.noname-c con 0x7feb640b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:30.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb640aa940 0x7feb64137d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient(hunting): picked mon.noname-b con 0x7feb640aa940 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb640abab0 con 0x7feb640b3b30 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb64138250 con 0x7feb640aa940 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb64138410 con 0x7feb640b4680 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6907c640 10 monclient(hunting): authenticate will time out at 1442.652989s 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb62d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b3b30 0x7feb6413a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:30.318 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b4680 0x7feb640ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.309+0000 7feb6359e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb640aa940 0x7feb64137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb62d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b3b30 0x7feb6413a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.162:54136/0 (socket says 172.21.15.162:54136) 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6359e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb640aa940 0x7feb64137d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:49512/0 (socket says 172.21.15.162:49512) 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb62d9d640 1 -- 172.21.15.162:0/3786853904 learned_addr learned my addr 172.21.15.162:0/3786853904 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb62d9d640 10 monclient(hunting): get_auth_request con 0x7feb640b3b30 auth_method 0 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb62d9d640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6359e640 10 monclient(hunting): get_auth_request con 0x7feb640aa940 auth_method 0 2024-01-27T03:42:30.319 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6359e640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6359e640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6259c640 10 monclient(hunting): get_auth_request con 0x7feb640b4680 auth_method 0 2024-01-27T03:42:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6259c640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:30.320 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.310+0000 7feb6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-01-27T03:42:30.321 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 1 -- 172.21.15.162:0/3786853904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b4680 msgr2=0x7feb640ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 1 --2- 172.21.15.162:0/3786853904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b4680 0x7feb640ab3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 1 -- 172.21.15.162:0/3786853904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b3b30 msgr2=0x7feb6413a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 1 --2- 172.21.15.162:0/3786853904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b3b30 0x7feb6413a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 10 monclient: _finish_hunting 0 2024-01-27T03:42:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 1 monclient: found mon.noname-b 2024-01-27T03:42:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:42:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.313+0000 7feb6359e640 1 -- 172.21.15.162:0/3786853904 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb54009400 con 0x7feb640aa940 2024-01-27T03:42:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.316+0000 7feb6359e640 1 --2- 172.21.15.162:0/3786853904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb640aa940 0x7feb64137d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00efc0 tx=0x7feb5c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.316+0000 7feb4b7fe640 10 client.?.objecter ms_handle_connect 0x7feb640aa940 2024-01-27T03:42:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.316+0000 7feb4b7fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 1 -- 172.21.15.162:0/3786853904 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c009c00 con 0x7feb640aa940 2024-01-27T03:42:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 10 monclient: dump: 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:30.326 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 10 monclient: _finish_auth 0 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 10 monclient: _check_auth_tickets 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 1 -- 172.21.15.162:0/3786853904 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5c00f040 con 0x7feb640aa940 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb6907c640 5 monclient: authenticate success, global_id 4138 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb6907c640 10 monclient: _renew_subs 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb6907c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb6907c640 1 -- 172.21.15.162:0/3786853904 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb641386f0 con 0x7feb640aa940 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 1 -- 172.21.15.162:0/3786853904 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c0146c0 con 0x7feb640aa940 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb4b7fe640 10 monclient: dump: 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-01-27T03:42:30.327 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:30.328 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:30.328 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:30.328 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:30.328 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-01-27T03:42:30.328 INFO:teuthology.orchestra.run.smithi162.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:30.328 INFO:teuthology.orchestra.run.smithi162.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:30.328 INFO:teuthology.orchestra.run.smithi162.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:30.328 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T03:42:30.328 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb60d99640 4 set_mon_vals no callback set 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb6907c640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb6907c640 10 monclient: _renew_subs 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb6907c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb6907c640 1 -- 172.21.15.162:0/3786853904 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb641da8b0 con 0x7feb640aa940 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.320+0000 7feb6907c640 1 librados: init done 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.321+0000 7feb6907c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.321+0000 7feb6907c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.321+0000 7feb6907c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.321+0000 7feb6907c640 1 -- 172.21.15.162:0/3786853904 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb640ab920 con 0x7feb640aa940 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.323+0000 7feb4b7fe640 1 -- 172.21.15.162:0/3786853904 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7feb5c014bb0 con 0x7feb640aa940 2024-01-27T03:42:30.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.323+0000 7feb4b7fe640 1 -- 172.21.15.162:0/3786853904 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7feb5c013a40 con 0x7feb640aa940 2024-01-27T03:42:30.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.323+0000 7feb4b7fe640 10 client.4138.objecter ms_dispatch 0x7feb64000b90 osd_map(9..9 src has 1..9) v4 2024-01-27T03:42:30.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.323+0000 7feb4b7fe640 3 client.4138.objecter handle_osd_map got epochs [9,9] > 0 2024-01-27T03:42:30.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.323+0000 7feb4b7fe640 3 client.4138.objecter handle_osd_map decoding full epoch 9 2024-01-27T03:42:30.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.323+0000 7feb4b7fe640 20 client.4138.objecter dump_active .. 0 homeless 2024-01-27T03:42:30.335 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:30.335+0000 7f8733841040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T03:42:30.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.422+0000 7feb4b7fe640 1 -- 172.21.15.162:0/3786853904 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7feb5c03bdb0 con 0x7feb640aa940 2024-01-27T03:42:30.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.423+0000 7feb4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:30.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.423+0000 7feb4b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:30.439 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:30.439+0000 7f8733841040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T03:42:30.456 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:30.455+0000 7f1407c0f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-27T03:42:30.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.494+0000 7feb6907c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7b8b531d-33ed-4bd5-8b97-d5ff4b210c17", "id": 7}] 2024-01-27T03:42:30.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.494+0000 7feb6907c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7b8b531d-33ed-4bd5-8b97-d5ff4b210c17", "id": 7}] 2024-01-27T03:42:30.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.494+0000 7feb6907c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:30.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.494+0000 7feb6907c640 1 -- 172.21.15.162:0/3786853904 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7b8b531d-33ed-4bd5-8b97-d5ff4b210c17", "id": 7} v 0) v1 -- 0x7feb6414cab0 con 0x7feb640aa940 2024-01-27T03:42:30.744 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:30.744+0000 7f1407c0f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-27T03:42:30.797 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:30.797+0000 7f1407c0f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-27T03:42:30.852 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:30.852+0000 7feb4b7fe640 1 -- 172.21.15.162:0/3786853904 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mgrmap(e 2) v1 ==== 44650+0+0 (secure 0 0 0) 0x7feb5c013c00 con 0x7feb640aa940 2024-01-27T03:42:30.904 INFO:tasks.ceph.mgr.y.smithi110.stderr:2024-01-27T03:42:30.903+0000 7f1407c0f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-27T03:42:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.181+0000 7feb4b7fe640 1 -- 172.21.15.162:0/3786853904 <== mon.1 v2:172.21.15.162:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7b8b531d-33ed-4bd5-8b97-d5ff4b210c17", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7feb5c018020 con 0x7feb640aa940 2024-01-27T03:42:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.181+0000 7feb4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7b8b531d-33ed-4bd5-8b97-d5ff4b210c17", "id": 7}] 2024-01-27T03:42:31.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.181+0000 7feb4b7fe640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:31.182 INFO:teuthology.orchestra.run.smithi162.stdout:7 2024-01-27T03:42:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 10 librados: watch_flush enter 2024-01-27T03:42:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 10 librados: watch_flush exit 2024-01-27T03:42:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 20 client.4138.objecter shutdown clearing up homeless session... 2024-01-27T03:42:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 10 client.4138.objecter successfully canceled tick 2024-01-27T03:42:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 10 monclient: shutdown 2024-01-27T03:42:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 -- 172.21.15.162:0/3786853904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb640aa940 msgr2=0x7feb64137d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:31.183 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 --2- 172.21.15.162:0/3786853904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb640aa940 0x7feb64137d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00efc0 tx=0x7feb5c00c490 comp rx=0 tx=0).stop 2024-01-27T03:42:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 -- 172.21.15.162:0/3786853904 shutdown_connections 2024-01-27T03:42:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 --2- 172.21.15.162:0/3786853904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb640b4680 0x7feb640ab3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 --2- 172.21.15.162:0/3786853904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb640aa940 0x7feb64137d10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 --2- 172.21.15.162:0/3786853904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb640b3b30 0x7feb6413a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 -- 172.21.15.162:0/3786853904 >> 172.21.15.162:0/3786853904 conn(0x7feb64066490 msgr2=0x7feb6413f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 -- 172.21.15.162:0/3786853904 shutdown_connections 2024-01-27T03:42:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 -- 172.21.15.162:0/3786853904 wait complete. 2024-01-27T03:42:31.184 INFO:teuthology.orchestra.run.smithi162.stderr:2024-01-27T03:42:31.182+0000 7feb6907c640 1 librados: shutdown 2024-01-27T03:42:31.193 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-27T03:42:31.193 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-27T03:42:31.193 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-01-27T03:42:31.195 INFO:tasks.ceph.osd.0:Started 2024-01-27T03:42:31.195 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-27T03:42:31.195 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-27T03:42:31.196 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-01-27T03:42:31.198 INFO:tasks.ceph.osd.1:Started 2024-01-27T03:42:31.198 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-27T03:42:31.198 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-27T03:42:31.198 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-01-27T03:42:31.200 INFO:tasks.ceph.osd.2:Started 2024-01-27T03:42:31.200 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-27T03:42:31.200 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-27T03:42:31.200 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-01-27T03:42:31.202 INFO:tasks.ceph.osd.3:Started 2024-01-27T03:42:31.202 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-27T03:42:31.203 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-27T03:42:31.203 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-01-27T03:42:31.205 INFO:tasks.ceph.osd.4:Started 2024-01-27T03:42:31.206 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-27T03:42:31.206 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-27T03:42:31.206 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-01-27T03:42:31.208 INFO:tasks.ceph.osd.5:Started 2024-01-27T03:42:31.208 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-27T03:42:31.208 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-27T03:42:31.209 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-01-27T03:42:31.210 INFO:tasks.ceph.osd.6:Started 2024-01-27T03:42:31.211 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-27T03:42:31.211 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-27T03:42:31.211 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-01-27T03:42:31.213 INFO:tasks.ceph.osd.7:Started 2024-01-27T03:42:31.213 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T03:42:31.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.362+0000 7f7bf9768640 10 monclient: get_monmap_and_config 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.362+0000 7f7bf9768640 10 monclient: build_initial_monmap 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.362+0000 7f7bf9768640 10 monclient: monmap: 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:42:31.363958+0000 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:42:31.363958+0000 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:31.365 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:31.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.363+0000 7f7bf9768640 1 Processor -- start 2024-01-27T03:42:31.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.363+0000 7f7bf9768640 1 -- start start 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.363+0000 7f7bf9768640 10 monclient: init 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7bf40b4680 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7bf40b4a60 0x7f7bf40b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 10 monclient(hunting): picked mon.noname-c con 0x7f7bf40b4a60 addr v2:172.21.15.162:3300/0 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b3b30 0x7f7bf40b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 10 monclient(hunting): picked mon.noname-b con 0x7f7bf40b3b30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf40afbe0 con 0x7f7bf40b4680 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf4142240 con 0x7f7bf40b4a60 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf40b0890 con 0x7f7bf40b3b30 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:31.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.364+0000 7f7bf9768640 10 monclient(hunting): authenticate will time out at 1434.718812s 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.365+0000 7f7bf37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b3b30 0x7f7bf40b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.365+0000 7f7bf37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b3b30 0x7f7bf40b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:34910/0 (socket says 172.21.15.110:34910) 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.365+0000 7f7bf37fe640 1 -- 172.21.15.110:0/2914125485 learned_addr learned my addr 172.21.15.110:0/2914125485 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.365+0000 7f7bf37fe640 10 monclient(hunting): get_auth_request con 0x7f7bf40b3b30 auth_method 0 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.365+0000 7f7bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.365+0000 7f7bf37fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.365+0000 7f7bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.366+0000 7f7bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.366+0000 7f7bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:31.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.366+0000 7f7bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:31.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.368+0000 7f7bebfff640 1 --2- 172.21.15.110:0/2914125485 >> v2:172.21.15.162:3300/0 conn(0x7f7bf40b4a60 0x7f7bf40b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.368+0000 7f7bebfff640 10 monclient(hunting): get_auth_request con 0x7f7bf40b4a60 auth_method 0 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.368+0000 7f7bebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.368+0000 7f7bebfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.368+0000 7f7bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.369+0000 7f7bf37fe640 10 monclient(hunting): handle_auth_done global_id 4160 payload 274 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.369+0000 7f7bf37fe640 1 -- 172.21.15.110:0/2914125485 >> v2:172.21.15.162:3300/0 conn(0x7f7bf40b4a60 msgr2=0x7f7bf40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.369+0000 7f7bf37fe640 1 --2- 172.21.15.110:0/2914125485 >> v2:172.21.15.162:3300/0 conn(0x7f7bf40b4a60 0x7f7bf40b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.369+0000 7f7bf37fe640 1 -- 172.21.15.110:0/2914125485 >> v1:172.21.15.162:6789/0 conn(0x7f7bf40b4680 legacy=0x7f7bf4142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.369+0000 7f7bf37fe640 10 monclient: _finish_hunting 0 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.369+0000 7f7bf37fe640 1 monclient: found mon.noname-b 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.369+0000 7f7bf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.369+0000 7f7bf37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:42:31.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.369+0000 7f7bf37fe640 1 -- 172.21.15.110:0/2914125485 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf40b0b00 con 0x7f7bf40b3b30 2024-01-27T03:42:31.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.374+0000 7f7bf37fe640 1 --2- 172.21.15.110:0/2914125485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b3b30 0x7f7bf40b19b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009e50 tx=0x7f7bec02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c985000b4f7b0360 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:31.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.442+0000 7f7bf27fc640 1 -- 172.21.15.110:0/2914125485 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec02c9e0 con 0x7f7bf40b3b30 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.442+0000 7f7bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.442+0000 7f7bf27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.442+0000 7f7bf27fc640 10 monclient: dump: 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:31.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.442+0000 7f7bf27fc640 10 monclient: _finish_auth 0 2024-01-27T03:42:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.442+0000 7f7bf27fc640 10 monclient: _check_auth_tickets 2024-01-27T03:42:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.442+0000 7f7bf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.442+0000 7f7bf9768640 5 monclient: authenticate success, global_id 4160 2024-01-27T03:42:31.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.442+0000 7f7bf9768640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.446+0000 7f7bf27fc640 1 -- 172.21.15.110:0/2914125485 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec02cb40 con 0x7f7bf40b3b30 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.446+0000 7f7bf27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.446+0000 7f7bf27fc640 1 -- 172.21.15.110:0/2914125485 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0343f0 con 0x7f7bf40b3b30 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.446+0000 7f7bf9768640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.446+0000 7f7bf9768640 4 set_mon_vals no callback set 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.446+0000 7f7bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.446+0000 7f7bf27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.446+0000 7f7bf27fc640 10 monclient: dump: 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:42:31.447 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:31.448 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:31.448 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 10 monclient: shutdown 2024-01-27T03:42:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 1 -- 172.21.15.110:0/2914125485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b3b30 msgr2=0x7f7bf40b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 1 --2- 172.21.15.110:0/2914125485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b3b30 0x7f7bf40b19b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009e50 tx=0x7f7bec02be40 comp rx=0 tx=0).stop 2024-01-27T03:42:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 1 -- 172.21.15.110:0/2914125485 shutdown_connections 2024-01-27T03:42:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 1 --2- 172.21.15.110:0/2914125485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b3b30 0x7f7bf40b19b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 1 --2- 172.21.15.110:0/2914125485 >> v2:172.21.15.162:3300/0 conn(0x7f7bf40b4a60 0x7f7bf40b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 1 -- 172.21.15.110:0/2914125485 >> 172.21.15.110:0/2914125485 conn(0x7f7bf4066490 msgr2=0x7f7bf40668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:31.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 1 -- 172.21.15.110:0/2914125485 shutdown_connections 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.447+0000 7f7bf9768640 1 -- 172.21.15.110:0/2914125485 wait complete. 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.448+0000 7f7bf9768640 10 monclient: build_initial_monmap 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.448+0000 7f7bf9768640 10 monclient: monmap: 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:31.450 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.449+0000 7f7bf9768640 1 librados: starting msgr at 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.449+0000 7f7bf9768640 1 librados: starting objecter 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.449+0000 7f7bf9768640 1 Processor -- start 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.449+0000 7f7bf9768640 1 -- start start 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.449+0000 7f7bf9768640 1 librados: setting wanted keys 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.449+0000 7f7bf9768640 1 librados: calling monclient init 2024-01-27T03:42:31.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.449+0000 7f7bf9768640 10 monclient: init 2024-01-27T03:42:31.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.450+0000 7f7bf9768640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:31.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.450+0000 7f7bf9768640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:42:31.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.450+0000 7f7bf9768640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bf40b3b30 0x7f7bf413dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:31.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.450+0000 7f7bf9768640 10 monclient(hunting): picked mon.noname-b con 0x7f7bf40b3b30 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:31.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.450+0000 7f7bf9768640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b4680 0x7f7bf406e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.450+0000 7f7bf9768640 10 monclient(hunting): picked mon.noname-c con 0x7f7bf40b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.450+0000 7f7bf9768640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bf40b7be0 0x7f7bf40b8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.450+0000 7f7bf9768640 10 monclient(hunting): picked mon.noname-a con 0x7f7bf40b7be0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf9768640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf9768640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf406de50 con 0x7f7bf40b7be0 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf9768640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf9768640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf406e050 con 0x7f7bf40b3b30 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bf40b3b30 0x7f7bf413dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf9768640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf9768640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf406e330 con 0x7f7bf40b4680 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf9768640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf9768640 10 monclient(hunting): authenticate will time out at 1434.805425s 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bf40b3b30 0x7f7bf413dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:51756/0 (socket says 172.21.15.110:51756) 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf2ffd640 1 -- 172.21.15.110:0/4268684629 learned_addr learned my addr 172.21.15.110:0/4268684629 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:42:31.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf2ffd640 10 monclient(hunting): get_auth_request con 0x7f7bf40b3b30 auth_method 0 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.451+0000 7f7bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.452+0000 7f7bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.452+0000 7f7bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.452+0000 7f7bf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.453+0000 7f7bebfff640 1 --2- 172.21.15.110:0/4268684629 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b4680 0x7f7bf406e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.453+0000 7f7bebfff640 10 monclient(hunting): get_auth_request con 0x7f7bf40b4680 auth_method 0 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.453+0000 7f7bebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.453+0000 7f7bebfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:31.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.453+0000 7f7bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:31.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.454+0000 7f7bebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:31.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.454+0000 7f7bebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:31.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.454+0000 7f7bebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:31.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 10 monclient(hunting): handle_auth_done global_id 4162 payload 824 2024-01-27T03:42:31.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 1 -- 172.21.15.110:0/4268684629 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b4680 msgr2=0x7f7bf406e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:31.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 1 --2- 172.21.15.110:0/4268684629 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b4680 0x7f7bf406e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 1 -- 172.21.15.110:0/4268684629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bf40b7be0 msgr2=0x7f7bf40b8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 1 --2- 172.21.15.110:0/4268684629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bf40b7be0 0x7f7bf40b8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 10 monclient: _finish_hunting 0 2024-01-27T03:42:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 1 monclient: found mon.noname-b 2024-01-27T03:42:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:42:31.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.456+0000 7f7bf2ffd640 1 -- 172.21.15.110:0/4268684629 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec009b30 con 0x7f7bf40b3b30 2024-01-27T03:42:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.460+0000 7f7bf2ffd640 1 --2- 172.21.15.110:0/4268684629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bf40b3b30 0x7f7bf413dea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7be400c430 tx=0x7f7be400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.460+0000 7f7beb7fe640 10 client.?.objecter ms_handle_connect 0x7f7bf40b3b30 2024-01-27T03:42:31.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.460+0000 7f7beb7fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:31.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 1 -- 172.21.15.110:0/4268684629 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4011790 con 0x7f7bf40b3b30 2024-01-27T03:42:31.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:31.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:31.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 10 monclient: dump: 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 10 monclient: _finish_auth 0 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 10 monclient: _check_auth_tickets 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 1 -- 172.21.15.110:0/4268684629 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be40118f0 con 0x7f7bf40b3b30 2024-01-27T03:42:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 1 -- 172.21.15.110:0/4268684629 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4011ce0 con 0x7f7bf40b3b30 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf9768640 5 monclient: authenticate success, global_id 4162 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7beb7fe640 10 monclient: dump: 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf9768640 10 monclient: _renew_subs 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf9768640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf9768640 1 -- 172.21.15.110:0/4268684629 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf40b76a0 con 0x7f7bf40b3b30 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf1ffb640 4 set_mon_vals no callback set 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf9768640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf9768640 10 monclient: _renew_subs 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf9768640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf9768640 1 -- 172.21.15.110:0/4268684629 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf40b8840 con 0x7f7bf40b3b30 2024-01-27T03:42:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.465+0000 7f7bf9768640 1 librados: init done 2024-01-27T03:42:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.466+0000 7f7be97fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.466+0000 7f7be97fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.466+0000 7f7be97fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.466+0000 7f7be97fa640 1 -- 172.21.15.110:0/4268684629 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf406e190 con 0x7f7bf40b3b30 2024-01-27T03:42:31.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.652+0000 7f7beb7fe640 1 -- 172.21.15.110:0/4268684629 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 2) v1 ==== 44650+0+0 (secure 0 0 0) 0x7f7be4026430 con 0x7f7bf40b3b30 2024-01-27T03:42:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.656+0000 7f7beb7fe640 1 -- 172.21.15.110:0/4268684629 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7be401b3c0 con 0x7f7bf40b3b30 2024-01-27T03:42:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.656+0000 7f7beb7fe640 10 client.4162.objecter ms_dispatch 0x7f7bf4000b90 osd_map(10..10 src has 1..10) v4 2024-01-27T03:42:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.656+0000 7f7beb7fe640 3 client.4162.objecter handle_osd_map got epochs [10,10] > 0 2024-01-27T03:42:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.656+0000 7f7beb7fe640 3 client.4162.objecter handle_osd_map decoding full epoch 10 2024-01-27T03:42:31.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.656+0000 7f7beb7fe640 20 client.4162.objecter dump_active .. 0 homeless 2024-01-27T03:42:31.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.663+0000 7f7beb7fe640 1 -- 172.21.15.110:0/4268684629 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73119 (secure 0 0 0) 0x7f7be4011e40 con 0x7f7bf40b3b30 2024-01-27T03:42:31.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.663+0000 7f7beb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:31.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.663+0000 7f7beb7fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.786+0000 7f7be97fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.786+0000 7f7be97fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.786+0000 7f7be97fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:31.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.786+0000 7f7be97fa640 1 -- 172.21.15.110:0/4268684629 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bf406e190 con 0x7f7bf40b3b30 2024-01-27T03:42:31.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.863+0000 7f7beb7fe640 1 -- 172.21.15.110:0/4268684629 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f7bf406e190 con 0x7f7bf40b3b30 2024-01-27T03:42:31.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.863+0000 7f7beb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:31.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.863+0000 7f7beb7fe640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:31.865 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:42:31.866 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":10,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:42:30.608380+0000","last_up_change":"0.000000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:42:31.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 10 librados: watch_flush enter 2024-01-27T03:42:31.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 10 librados: watch_flush exit 2024-01-27T03:42:31.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 20 client.4162.objecter shutdown clearing up homeless session... 2024-01-27T03:42:31.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 10 client.4162.objecter successfully canceled tick 2024-01-27T03:42:31.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 10 monclient: shutdown 2024-01-27T03:42:31.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:31.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 -- 172.21.15.110:0/4268684629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bf40b3b30 msgr2=0x7f7bf413dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:31.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 --2- 172.21.15.110:0/4268684629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bf40b3b30 0x7f7bf413dea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7be400c430 tx=0x7f7be400c8f0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 -- 172.21.15.110:0/4268684629 shutdown_connections 2024-01-27T03:42:31.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 --2- 172.21.15.110:0/4268684629 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7bf40b4680 0x7f7bf406e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 --2- 172.21.15.110:0/4268684629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7bf40b7be0 0x7f7bf40b8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 --2- 172.21.15.110:0/4268684629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bf40b3b30 0x7f7bf413dea0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:31.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 -- 172.21.15.110:0/4268684629 >> 172.21.15.110:0/4268684629 conn(0x7f7bf4066490 msgr2=0x7f7bf413b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:31.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 -- 172.21.15.110:0/4268684629 shutdown_connections 2024-01-27T03:42:31.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 -- 172.21.15.110:0/4268684629 wait complete. 2024-01-27T03:42:31.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:42:31.865+0000 7f7be97fa640 1 librados: shutdown 2024-01-27T03:42:31.881 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-27T03:42:31.882 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-27T03:42:32.688 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-27T03:42:32.815 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:32.815+0000 7f86d4c95640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.847+0000 7fb8453d5640 10 monclient: get_monmap_and_config 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.847+0000 7fb8453d5640 10 monclient: build_initial_monmap 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.847+0000 7fb8453d5640 10 monclient: monmap: 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:42:32.849040+0000 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:42:32.849040+0000 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:32.849 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:32.850 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:32.850 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:32.850 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.848+0000 7fb8453d5640 1 Processor -- start 2024-01-27T03:42:32.850 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.848+0000 7fb8453d5640 1 -- start start 2024-01-27T03:42:32.850 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.849+0000 7fb8453d5640 10 monclient: init 2024-01-27T03:42:32.850 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.849+0000 7fb8453d5640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:32.850 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.849+0000 7fb8453d5640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.849+0000 7fb8453d5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400b4c80 0x7fb84013f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.850+0000 7fb8453d5640 10 monclient(hunting): picked mon.noname-b con 0x7fb8400b4c80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.850+0000 7fb8453d5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8400a9e20 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.850+0000 7fb8453d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb840142770 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.850+0000 7fb8453d5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8400a99d0 con 0x7fb840142770 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.850+0000 7fb8453d5640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8400b3ed0 con 0x7fb8400a9e20 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.850+0000 7fb8453d5640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.850+0000 7fb8453d5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400b5060 con 0x7fb8400b4c80 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.850+0000 7fb8453d5640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:32.851 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.850+0000 7fb8453d5640 10 monclient(hunting): authenticate will time out at 1436.204025s 2024-01-27T03:42:32.852 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.851+0000 7fb83e7fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb8400a9e20 0x7fb840142060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:51114/0 (socket says 172.21.15.110:51114) 2024-01-27T03:42:32.852 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.851+0000 7fb83e7fc640 1 -- 172.21.15.110:0/1233515512 learned_addr learned my addr 172.21.15.110:0/1233515512 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:42:32.892 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.891+0000 7fb83effd640 1 --2- 172.21.15.110:0/1233515512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400b4c80 0x7fb84013f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:32.892 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.892+0000 7fb83effd640 10 monclient(hunting): get_auth_request con 0x7fb8400b4c80 auth_method 0 2024-01-27T03:42:32.893 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.892+0000 7fb83effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:32.893 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.892+0000 7fb83effd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:32.893 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.892+0000 7fb83effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:32.893 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.892+0000 7fb83effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:32.893 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.892+0000 7fb83effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:32.893 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.892+0000 7fb83effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:32.906 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T03:42:32.906+0000 7f86d4c95640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-27T03:42:32.911 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.910+0000 7fb83effd640 10 monclient(hunting): handle_auth_done global_id 4166 payload 274 2024-01-27T03:42:32.911 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.910+0000 7fb83effd640 1 -- 172.21.15.110:0/1233515512 >> v1:172.21.15.162:6789/0 conn(0x7fb8400a9e20 legacy=0x7fb840142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:32.911 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.910+0000 7fb83effd640 1 -- 172.21.15.110:0/1233515512 >> v1:172.21.15.110:6789/0 conn(0x7fb840142770 legacy=0x7fb84014c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:32.911 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.910+0000 7fb83effd640 10 monclient: _finish_hunting 0 2024-01-27T03:42:32.911 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.910+0000 7fb83effd640 1 monclient: found mon.noname-b 2024-01-27T03:42:32.911 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.910+0000 7fb83effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:32.911 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.910+0000 7fb83effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:42:32.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.910+0000 7fb83effd640 1 -- 172.21.15.110:0/1233515512 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8400b3b60 con 0x7fb8400b4c80 2024-01-27T03:42:32.914 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.913+0000 7fb83effd640 1 --2- 172.21.15.110:0/1233515512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400b4c80 0x7fb84013f300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb8300097d0 tx=0x7fb83002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a559e6b84d0f8ffc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:32.917 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.916+0000 7fb83dffb640 1 -- 172.21.15.110:0/1233515512 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb830030800 con 0x7fb8400b4c80 2024-01-27T03:42:32.917 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.916+0000 7fb83dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.917+0000 7fb83dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.917+0000 7fb83dffb640 10 monclient: dump: 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.917+0000 7fb83dffb640 10 monclient: _finish_auth 0 2024-01-27T03:42:32.918 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.917+0000 7fb83dffb640 10 monclient: _check_auth_tickets 2024-01-27T03:42:32.919 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.917+0000 7fb83dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:32.919 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.917+0000 7fb8453d5640 5 monclient: authenticate success, global_id 4166 2024-01-27T03:42:32.919 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.917+0000 7fb8453d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:42:32.920 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.919+0000 7fb83dffb640 1 -- 172.21.15.110:0/1233515512 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb830030960 con 0x7fb8400b4c80 2024-01-27T03:42:32.920 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.919+0000 7fb83dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:32.920 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.919+0000 7fb83dffb640 1 -- 172.21.15.110:0/1233515512 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb830030cf0 con 0x7fb8400b4c80 2024-01-27T03:42:32.920 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.919+0000 7fb83dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:32.920 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.919+0000 7fb83dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:32.920 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.919+0000 7fb83dffb640 10 monclient: dump: 2024-01-27T03:42:32.920 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:32.920 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.919+0000 7fb8453d5640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.919+0000 7fb8453d5640 4 set_mon_vals no callback set 2024-01-27T03:42:32.921 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.920+0000 7fb8453d5640 10 monclient: shutdown 2024-01-27T03:42:32.922 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.920+0000 7fb8453d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:32.922 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.920+0000 7fb8453d5640 1 -- 172.21.15.110:0/1233515512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400b4c80 msgr2=0x7fb84013f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:32.922 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.920+0000 7fb8453d5640 1 --2- 172.21.15.110:0/1233515512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400b4c80 0x7fb84013f300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb8300097d0 tx=0x7fb83002b940 comp rx=0 tx=0).stop 2024-01-27T03:42:32.922 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.921+0000 7fb8453d5640 1 -- 172.21.15.110:0/1233515512 shutdown_connections 2024-01-27T03:42:32.922 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.921+0000 7fb8453d5640 1 --2- 172.21.15.110:0/1233515512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400b4c80 0x7fb84013f300 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:32.922 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.921+0000 7fb8453d5640 1 -- 172.21.15.110:0/1233515512 >> 172.21.15.110:0/1233515512 conn(0x7fb840071f90 msgr2=0x7fb8400723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:32.922 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.921+0000 7fb8453d5640 1 -- 172.21.15.110:0/1233515512 shutdown_connections 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.921+0000 7fb8453d5640 1 -- 172.21.15.110:0/1233515512 wait complete. 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.922+0000 7fb8453d5640 10 monclient: build_initial_monmap 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.922+0000 7fb8453d5640 10 monclient: monmap: 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:32.923 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:32.924 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:32.924 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.922+0000 7fb8453d5640 1 librados: starting msgr at 2024-01-27T03:42:32.924 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.922+0000 7fb8453d5640 1 librados: starting objecter 2024-01-27T03:42:32.924 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.923+0000 7fb8453d5640 1 Processor -- start 2024-01-27T03:42:32.924 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.923+0000 7fb8453d5640 1 -- start start 2024-01-27T03:42:32.925 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.923+0000 7fb8453d5640 1 librados: setting wanted keys 2024-01-27T03:42:32.925 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.923+0000 7fb8453d5640 1 librados: calling monclient init 2024-01-27T03:42:32.925 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.923+0000 7fb8453d5640 10 monclient: init 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8400b4c80 0x7fb84013adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient(hunting): picked mon.noname-b con 0x7fb8400b4c80 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb840142770 0x7fb8400af140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient(hunting): picked mon.noname-a con 0x7fb840142770 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400ae270 0x7fb8400b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient(hunting): picked mon.noname-c con 0x7fb8400ae270 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400aecf0 con 0x7fb840142770 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:32.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400af680 con 0x7fb8400b4c80 2024-01-27T03:42:32.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:32.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400b0660 con 0x7fb8400ae270 2024-01-27T03:42:32.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:32.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.924+0000 7fb8453d5640 10 monclient(hunting): authenticate will time out at 1436.278452s 2024-01-27T03:42:32.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400ae270 0x7fb8400b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:32.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8400b4c80 0x7fb84013adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400ae270 0x7fb8400b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:34944/0 (socket says 172.21.15.110:34944) 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83f7fe640 1 -- 172.21.15.110:0/3314107457 learned_addr learned my addr 172.21.15.110:0/3314107457 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83f7fe640 10 monclient(hunting): get_auth_request con 0x7fb8400ae270 auth_method 0 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83effd640 10 monclient(hunting): get_auth_request con 0x7fb8400b4c80 auth_method 0 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83effd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.925+0000 7fb83effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.926+0000 7fb83effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.926+0000 7fb83effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:32.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.926+0000 7fb83effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:32.930 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.926+0000 7fb83f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:32.930 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.926+0000 7fb83f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:32.930 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.926+0000 7fb83f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:32.932 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 10 monclient(hunting): handle_auth_done global_id 4165 payload 824 2024-01-27T03:42:32.933 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 1 -- 172.21.15.110:0/3314107457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400ae270 msgr2=0x7fb8400b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:32.933 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400ae270 0x7fb8400b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:32.933 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 1 -- 172.21.15.110:0/3314107457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb840142770 msgr2=0x7fb8400af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:32.933 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb840142770 0x7fb8400af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:32.933 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 10 monclient: _finish_hunting 0 2024-01-27T03:42:32.933 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 1 monclient: found mon.noname-b 2024-01-27T03:42:32.933 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:32.933 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:42:32.933 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.931+0000 7fb83effd640 1 -- 172.21.15.110:0/3314107457 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb830009400 con 0x7fb8400b4c80 2024-01-27T03:42:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.958+0000 7fb83effd640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8400b4c80 0x7fb84013adf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb830033000 tx=0x7fb830004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.959+0000 7fb8277fe640 10 client.?.objecter ms_handle_connect 0x7fb8400b4c80 2024-01-27T03:42:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:32.959+0000 7fb8277fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.064+0000 7fb8277fe640 1 -- 172.21.15.110:0/3314107457 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83003e040 con 0x7fb8400b4c80 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.064+0000 7fb8277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.064+0000 7fb8277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.064+0000 7fb8277fe640 10 monclient: dump: 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:33.068 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb8277fe640 10 monclient: _finish_auth 0 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb8277fe640 10 monclient: _check_auth_tickets 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb8277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb8277fe640 1 -- 172.21.15.110:0/3314107457 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb830039de0 con 0x7fb8400b4c80 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb8277fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb8277fe640 1 -- 172.21.15.110:0/3314107457 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb830044630 con 0x7fb8400b4c80 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb8277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb8277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:42:33.069 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb8277fe640 10 monclient: dump: 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.065+0000 7fb83d7fa640 4 set_mon_vals no callback set 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.066+0000 7fb8453d5640 5 monclient: authenticate success, global_id 4165 2024-01-27T03:42:33.070 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.066+0000 7fb8453d5640 10 monclient: _renew_subs 2024-01-27T03:42:33.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.066+0000 7fb8453d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:33.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.066+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8400ae720 con 0x7fb8400b4c80 2024-01-27T03:42:33.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.066+0000 7fb8453d5640 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:33.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.066+0000 7fb8453d5640 10 monclient: _renew_subs 2024-01-27T03:42:33.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.066+0000 7fb8453d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:33.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.066+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8400b11a0 con 0x7fb8400b4c80 2024-01-27T03:42:33.071 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.066+0000 7fb8453d5640 1 librados: init done 2024-01-27T03:42:33.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.067+0000 7fb8453d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:33.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.067+0000 7fb8453d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:33.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.067+0000 7fb8453d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:33.072 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.067+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84013fc20 con 0x7fb8400b4c80 2024-01-27T03:42:33.492 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.491+0000 7fb8277fe640 1 -- 172.21.15.110:0/3314107457 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 3) v1 ==== 89088+0+0 (secure 0 0 0) 0x7fb830044790 con 0x7fb8400b4c80 2024-01-27T03:42:33.493 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.491+0000 7fb8277fe640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb82c06c840 0x7fb82c06ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:33.493 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.492+0000 7fb83e7fc640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb82c06c840 0x7fb82c06ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:33.493 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.492+0000 7fb83e7fc640 10 monclient: get_auth_request con 0x7fb82c06c840 auth_method 0 2024-01-27T03:42:33.493 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.492+0000 7fb8277fe640 10 client.4165.objecter ms_handle_connect 0x7fb82c06c840 2024-01-27T03:42:33.493 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.492+0000 7fb83e7fc640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb82c06c840 0x7fb82c06ed00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb8400affd0 tx=0x7fb834007a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:33.516 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.515+0000 7fb8277fe640 1 -- 172.21.15.110:0/3314107457 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb83007af30 con 0x7fb8400b4c80 2024-01-27T03:42:33.516 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.515+0000 7fb8277fe640 10 client.4165.objecter ms_dispatch 0x7fb840000b90 osd_map(10..10 src has 1..10) v4 2024-01-27T03:42:33.516 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.515+0000 7fb8277fe640 3 client.4165.objecter handle_osd_map got epochs [10,10] > 0 2024-01-27T03:42:33.516 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.515+0000 7fb8277fe640 3 client.4165.objecter handle_osd_map decoding full epoch 10 2024-01-27T03:42:33.516 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.515+0000 7fb8277fe640 20 client.4165.objecter dump_active .. 0 homeless 2024-01-27T03:42:33.518 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.517+0000 7fb8277fe640 1 -- 172.21.15.110:0/3314107457 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fb8300771d0 con 0x7fb8400b4c80 2024-01-27T03:42:33.518 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.517+0000 7fb8277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:33.518 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.517+0000 7fb8277fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:33.879 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.877+0000 7fb8453d5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:33.879 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.878+0000 7fb8453d5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:33.879 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.878+0000 7fb8453d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:33.879 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.878+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8400b07a0 con 0x7fb8400b4c80 2024-01-27T03:42:33.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.899+0000 7fb8277fe640 1 -- 172.21.15.110:0/3314107457 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mgrmap(e 4) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fb8300775a0 con 0x7fb8400b4c80 2024-01-27T03:42:33.931 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.930+0000 7fb8277fe640 1 -- 172.21.15.110:0/3314107457 <== mon.1 v2:172.21.15.162:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fb830094d70 con 0x7fb8400b4c80 2024-01-27T03:42:33.931 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.930+0000 7fb8277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:33.931 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.930+0000 7fb8277fe640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:33.931 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-27T03:42:33.932 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:42:30.608380+0000","last_up_change":"0.000000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 10 librados: watch_flush enter 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 10 librados: watch_flush exit 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 20 client.4165.objecter shutdown clearing up homeless session... 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 10 client.4165.objecter successfully canceled tick 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb82c06c840 msgr2=0x7fb82c06ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb82c06c840 0x7fb82c06ed00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb8400affd0 tx=0x7fb834007a40 comp rx=0 tx=0).stop 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 10 monclient: shutdown 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8400b4c80 msgr2=0x7fb84013adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8400b4c80 0x7fb84013adf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb830033000 tx=0x7fb830004730 comp rx=0 tx=0).stop 2024-01-27T03:42:33.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 shutdown_connections 2024-01-27T03:42:33.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb82c06c840 0x7fb82c06ed00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:33.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8400ae270 0x7fb8400b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:33.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb840142770 0x7fb8400af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:33.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 --2- 172.21.15.110:0/3314107457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8400b4c80 0x7fb84013adf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:33.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 >> 172.21.15.110:0/3314107457 conn(0x7fb840071f90 msgr2=0x7fb840073f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:33.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 shutdown_connections 2024-01-27T03:42:33.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 -- 172.21.15.110:0/3314107457 wait complete. 2024-01-27T03:42:33.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:33.933+0000 7fb8453d5640 1 librados: shutdown 2024-01-27T03:42:33.949 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-27T03:42:38.660 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:07.389 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.726 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:07.455 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.728 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:07.453 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.776 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:07.499 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.794 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:07.520 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.827 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:07.545 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.838 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:07.560 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.873 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:07.604 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.889 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:07.606 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.937 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:07.667 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.972 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:07.700 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:38.977 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:07.685 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:39.026 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:07.741 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:39.038 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:07.746 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:39.042 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:07.771 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:39.087 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:07.803 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:39.267 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:42:39.261+0000 5dc2480 -1 Falling back to public interface 2024-01-27T03:42:39.348 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:42:39.339+0000 5dc2480 -1 Falling back to public interface 2024-01-27T03:42:39.398 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T03:42:39.390+0000 5dc2480 -1 Falling back to public interface 2024-01-27T03:42:39.435 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-27T03:42:39.430+0000 5dc2480 -1 Falling back to public interface 2024-01-27T03:42:39.494 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:42:39.487+0000 5dc2480 -1 Falling back to public interface 2024-01-27T03:42:39.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:42:39.576+0000 5dc2480 -1 Falling back to public interface 2024-01-27T03:42:39.623 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:42:39.618+0000 5dc2480 -1 Falling back to public interface 2024-01-27T03:42:39.652 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:42:39.644+0000 5dc2480 -1 Falling back to public interface 2024-01-27T03:42:39.781 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:08.511 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:39.794 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:08.520 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:39.893 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:08.611 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:39.928 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:08.650 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:39.948 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:08.678 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.057 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:08.765 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.073 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:08.801 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.135 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:08.850 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.147 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:08.876 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.162 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:08.887 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.261 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:08.976 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.309 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:09.031 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.337 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:09.067 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.428 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.136 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.462 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:09.191 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.464 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.193 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.509 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:09.224 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.525 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:09.250 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.574 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.290 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.622 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:09.344 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.691 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:09.421 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.733 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:09.462 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.778 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.506 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.781 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.490 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.814 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:09.540 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.835 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-27T03:42:40.858 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.576 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.864 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:09.580 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.888 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:09.610 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:40.924 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.924+0000 7f9e114f2640 10 monclient: get_monmap_and_config 2024-01-27T03:42:40.924 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.924+0000 7f9e114f2640 10 monclient: build_initial_monmap 2024-01-27T03:42:40.924 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.924+0000 7f9e114f2640 10 monclient: monmap: 2024-01-27T03:42:40.924 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:42:40.924 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:40.924 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:42:40.924515+0000 2024-01-27T03:42:40.925 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:42:40.924515+0000 2024-01-27T03:42:40.925 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:40.925 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:40.925 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:40.925 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:40.925 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:40.925 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:40.925 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:40.925 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:40.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 1 Processor -- start 2024-01-27T03:42:40.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 1 -- start start 2024-01-27T03:42:40.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient: init 2024-01-27T03:42:40.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:40.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e0c06c470 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9e0c0af120 0x7f9e0c0af560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient(hunting): picked mon.noname-c con 0x7f9e0c0af120 addr v2:172.21.15.162:3300/0 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e0c14aad0 0x7f9e0c14cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient(hunting): picked mon.noname-a con 0x7f9e0c14aad0 addr v2:172.21.15.110:3300/0 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0c0b0950 con 0x7f9e0c06c470 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0c06c850 con 0x7f9e0c14aad0 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0c0b0c30 con 0x7f9e0c0af120 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.925+0000 7f9e114f2640 10 monclient(hunting): authenticate will time out at 1444.278660s 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.926+0000 7f9e03fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9e0c0af120 0x7f9e0c0af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:40.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.926+0000 7f9e03fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9e0c0af120 0x7f9e0c0af560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:57350/0 (socket says 172.21.15.110:57350) 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.926+0000 7f9e0b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e0c14aad0 0x7f9e0c14cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.926+0000 7f9e03fff640 1 -- 172.21.15.110:0/1729822694 learned_addr learned my addr 172.21.15.110:0/1729822694 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.926+0000 7f9e03fff640 10 monclient(hunting): get_auth_request con 0x7f9e0c0af120 auth_method 0 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.926+0000 7f9e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.926+0000 7f9e03fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.926+0000 7f9e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e0b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e0c14aad0 auth_method 0 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e0b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:40.929 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:40.930 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.927+0000 7f9e0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:40.930 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.928+0000 7f9e03fff640 10 monclient(hunting): handle_auth_done global_id 4195 payload 274 2024-01-27T03:42:40.930 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.928+0000 7f9e03fff640 1 -- 172.21.15.110:0/1729822694 >> v2:172.21.15.110:3300/0 conn(0x7f9e0c14aad0 msgr2=0x7f9e0c14cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:40.930 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.928+0000 7f9e03fff640 1 --2- 172.21.15.110:0/1729822694 >> v2:172.21.15.110:3300/0 conn(0x7f9e0c14aad0 0x7f9e0c14cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:40.930 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.928+0000 7f9e03fff640 1 -- 172.21.15.110:0/1729822694 >> v1:172.21.15.110:6789/0 conn(0x7f9e0c06c470 legacy=0x7f9e0c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:40.931 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.928+0000 7f9e03fff640 10 monclient: _finish_hunting 0 2024-01-27T03:42:40.931 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.928+0000 7f9e03fff640 1 monclient: found mon.noname-c 2024-01-27T03:42:40.931 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.928+0000 7f9e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:40.931 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.928+0000 7f9e03fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:42:40.931 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.928+0000 7f9e03fff640 1 -- 172.21.15.110:0/1729822694 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c0b40a0 con 0x7f9e0c0af120 2024-01-27T03:42:40.931 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.930+0000 7f9e03fff640 1 --2- 172.21.15.110:0/1729822694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e0c0af120 0x7f9e0c0af560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0b5670 tx=0x7f9df402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=755de8727b19893b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:40.934 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 1 -- 172.21.15.110:0/1729822694 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4031870 con 0x7f9e0c0af120 2024-01-27T03:42:40.934 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 10 monclient: dump: 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 10 monclient: _finish_auth 0 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:40.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 1 -- 172.21.15.110:0/1729822694 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df40319d0 con 0x7f9e0c0af120 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 1 -- 172.21.15.110:0/1729822694 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4031d60 con 0x7f9e0c0af120 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e0a7fc640 10 monclient: dump: 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:40.936 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:40.937 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:40.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e114f2640 5 monclient: authenticate success, global_id 4195 2024-01-27T03:42:40.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e114f2640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:40.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.934+0000 7f9e114f2640 4 set_mon_vals no callback set 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 10 monclient: shutdown 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 1 -- 172.21.15.110:0/1729822694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e0c0af120 msgr2=0x7f9e0c0af560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 1 --2- 172.21.15.110:0/1729822694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e0c0af120 0x7f9e0c0af560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0b5670 tx=0x7f9df402b910 comp rx=0 tx=0).stop 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 1 -- 172.21.15.110:0/1729822694 shutdown_connections 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 1 --2- 172.21.15.110:0/1729822694 >> v2:172.21.15.110:3300/0 conn(0x7f9e0c14aad0 0x7f9e0c14cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 1 --2- 172.21.15.110:0/1729822694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e0c0af120 0x7f9e0c0af560 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 1 -- 172.21.15.110:0/1729822694 >> 172.21.15.110:0/1729822694 conn(0x7f9e0c0b7c90 msgr2=0x7f9e0c0b80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 1 -- 172.21.15.110:0/1729822694 shutdown_connections 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.935+0000 7f9e114f2640 1 -- 172.21.15.110:0/1729822694 wait complete. 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.936+0000 7f9e114f2640 10 monclient: build_initial_monmap 2024-01-27T03:42:40.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.936+0000 7f9e114f2640 10 monclient: monmap: 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.936+0000 7f9e114f2640 1 librados: starting msgr at 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.936+0000 7f9e114f2640 1 librados: starting objecter 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.936+0000 7f9e114f2640 1 Processor -- start 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.936+0000 7f9e114f2640 1 -- start start 2024-01-27T03:42:40.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.936+0000 7f9e114f2640 1 librados: setting wanted keys 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.936+0000 7f9e114f2640 1 librados: calling monclient init 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.936+0000 7f9e114f2640 10 monclient: init 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e0c06c470 0x7f9e0c1358c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient(hunting): picked mon.noname-b con 0x7f9e0c06c470 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e0c0af120 0x7f9e0c1d7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient(hunting): picked mon.noname-a con 0x7f9e0c0af120 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e0c14aad0 0x7f9e0c1d7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient(hunting): picked mon.noname-c con 0x7f9e0c14aad0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c1d7290 con 0x7f9e0c0af120 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:40.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c1d81a0 con 0x7f9e0c06c470 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c1371a0 con 0x7f9e0c14aad0 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e114f2640 10 monclient(hunting): authenticate will time out at 1444.290219s 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e03fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e0c0af120 0x7f9e0c1d7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e0b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e0c14aad0 0x7f9e0c1d7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e03fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e0c0af120 0x7f9e0c1d7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:35776/0 (socket says 172.21.15.110:35776) 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e03fff640 1 -- 172.21.15.110:0/1781813731 learned_addr learned my addr 172.21.15.110:0/1781813731 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e03fff640 10 monclient(hunting): get_auth_request con 0x7f9e0c0af120 auth_method 0 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e03fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.937+0000 7f9e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0affd640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e0c06c470 0x7f9e0c1358c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e0c14aad0 auth_method 0 2024-01-27T03:42:40.944 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0affd640 10 monclient(hunting): get_auth_request con 0x7f9e0c06c470 auth_method 0 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0affd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:40.945 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.938+0000 7f9e0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 1 -- 172.21.15.110:0/1781813731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e0c14aad0 msgr2=0x7f9e0c1d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e0c14aad0 0x7f9e0c1d7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 1 -- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e0c06c470 msgr2=0x7f9e0c1358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e0c06c470 0x7f9e0c1358c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 10 monclient: _finish_hunting 0 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 1 monclient: found mon.noname-a 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:40.946 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.940+0000 7f9e03fff640 1 -- 172.21.15.110:0/1781813731 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df40094d0 con 0x7f9e0c0af120 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.943+0000 7f9e03fff640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e0c0af120 0x7f9e0c1d7720 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009960 tx=0x7f9df40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.943+0000 7f9e037fe640 10 client.?.objecter ms_handle_connect 0x7f9e0c0af120 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.943+0000 7f9e037fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 1 -- 172.21.15.110:0/1781813731 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df403abe0 con 0x7f9e0c0af120 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 10 monclient: dump: 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:40.948 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 10 monclient: _finish_auth 0 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 10 monclient: _check_auth_tickets 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 1 -- 172.21.15.110:0/1781813731 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df403ad40 con 0x7f9e0c0af120 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 1 -- 172.21.15.110:0/1781813731 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4044630 con 0x7f9e0c0af120 2024-01-27T03:42:40.949 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e037fe640 10 monclient: dump: 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e114f2640 5 monclient: authenticate success, global_id 4197 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e09ffb640 4 set_mon_vals no callback set 2024-01-27T03:42:40.950 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e114f2640 10 monclient: _renew_subs 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e114f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0c137410 con 0x7f9e0c0af120 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e114f2640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e114f2640 10 monclient: _renew_subs 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e114f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c134d40 con 0x7f9e0c0af120 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.946+0000 7f9e114f2640 1 librados: init done 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.947+0000 7f9e114f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.947+0000 7f9e114f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.947+0000 7f9e114f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:40.951 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.947+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0c1d7400 con 0x7f9e0c0af120 2024-01-27T03:42:40.965 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.965+0000 7f9e037fe640 1 -- 172.21.15.110:0/1781813731 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f9df4044790 con 0x7f9e0c0af120 2024-01-27T03:42:40.966 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.966+0000 7f9e037fe640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f9dec06c990 0x7f9dec06ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:40.966 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.966+0000 7f9e0affd640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f9dec06c990 0x7f9dec06ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:40.966 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.966+0000 7f9e0affd640 10 monclient: get_auth_request con 0x7f9dec06c990 auth_method 0 2024-01-27T03:42:40.966 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.966+0000 7f9e037fe640 10 client.4197.objecter ms_handle_connect 0x7f9dec06c990 2024-01-27T03:42:40.966 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.966+0000 7f9e0affd640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f9dec06c990 0x7f9dec06ee50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc000f60 tx=0x7f9dfc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:40.967 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.967+0000 7f9e037fe640 1 -- 172.21.15.110:0/1781813731 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9df402d040 con 0x7f9e0c0af120 2024-01-27T03:42:40.967 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.967+0000 7f9e037fe640 10 client.4197.objecter ms_dispatch 0x7f9e0c000b90 osd_map(10..10 src has 1..10) v4 2024-01-27T03:42:40.967 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.967+0000 7f9e037fe640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2024-01-27T03:42:40.967 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.967+0000 7f9e037fe640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2024-01-27T03:42:40.967 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:40.967+0000 7f9e037fe640 20 client.4197.objecter dump_active .. 0 homeless 2024-01-27T03:42:40.985 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:09.714 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.010 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:09.739 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.046 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:09.775 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.076 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:09.784 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.101 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:09.827 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.126 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:09.844 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.151 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:09.874 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.158 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:09.875 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.201 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.201+0000 7f9e037fe640 1 -- 172.21.15.110:0/1781813731 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f9df4077380 con 0x7f9e0c0af120 2024-01-27T03:42:41.201 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.201+0000 7f9e037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:41.201 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.201+0000 7f9e037fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:41.278 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:10.007 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.278 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:10.008 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.310 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.039 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.370 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.078 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.384 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.384+0000 7f9e114f2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:41.384 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.384+0000 7f9e114f2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:41.384 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.384+0000 7f9e114f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:42:41.384 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.384+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e0c04f2a0 con 0x7f9e0c0af120 2024-01-27T03:42:41.389 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:10.115 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.399 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.117 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.419 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:10.141 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.428 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.428+0000 7f9e037fe640 1 -- 172.21.15.110:0/1781813731 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f9df407aa00 con 0x7f9e0c0af120 2024-01-27T03:42:41.429 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.428+0000 7f9e037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:41.429 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.428+0000 7f9e037fe640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:41.429 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-27T03:42:41.429 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:42:30.608380+0000","last_up_change":"0.000000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:42:41.431 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 10 librados: watch_flush enter 2024-01-27T03:42:41.431 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 10 librados: watch_flush exit 2024-01-27T03:42:41.431 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 20 client.4197.objecter shutdown clearing up homeless session... 2024-01-27T03:42:41.431 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 10 client.4197.objecter successfully canceled tick 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f9dec06c990 msgr2=0x7f9dec06ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f9dec06c990 0x7f9dec06ee50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc000f60 tx=0x7f9dfc009040 comp rx=0 tx=0).stop 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 10 monclient: shutdown 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e0c0af120 msgr2=0x7f9e0c1d7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e0c0af120 0x7f9e0c1d7720 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009960 tx=0x7f9df40046c0 comp rx=0 tx=0).stop 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 shutdown_connections 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f9dec06c990 0x7f9dec06ee50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e0c14aad0 0x7f9e0c1d7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e0c0af120 0x7f9e0c1d7720 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 --2- 172.21.15.110:0/1781813731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e0c06c470 0x7f9e0c1358c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.430+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 >> 172.21.15.110:0/1781813731 conn(0x7f9e0c0b7c90 msgr2=0x7f9e0c13ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:41.432 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.431+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 shutdown_connections 2024-01-27T03:42:41.433 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.431+0000 7f9e114f2640 1 -- 172.21.15.110:0/1781813731 wait complete. 2024-01-27T03:42:41.433 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:41.431+0000 7f9e114f2640 1 librados: shutdown 2024-01-27T03:42:41.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-27T03:42:41.469 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.185 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.552 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:10.281 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.552 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:10.282 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.581 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.310 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.666 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.374 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.669 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.387 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.677 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:10.403 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.696 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:10.419 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.759 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.475 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.818 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:10.548 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.823 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:10.553 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.846 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.575 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.936 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.654 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.963 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.671 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.965 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:10.687 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:41.965 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:10.691 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.053 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:10.768 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.084 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:10.813 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.110 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:10.839 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.113 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:10.843 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.200 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:10.918 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.229 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:10.952 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.251 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:10.959 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.261 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:10.987 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.344 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:11.060 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.375 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:11.104 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.407 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:11.137 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.541 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:11.249 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.554 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:11.283 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.611 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:11.340 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.633 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:11.348 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.685 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:11.403 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.703 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:11.426 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.742 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:11.460 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.758 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:11.481 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.783 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:11.508 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.840 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:11.565 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.864 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:11.592 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.918 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:11.646 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.924 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:11.654 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:42.979 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:11.709 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:43.137 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:11.845 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:43.200 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:11.908 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:43.269 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:11.984 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:43.324 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:12.039 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:48.049 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:16.772 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:48.160 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:16.886 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:48.247 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-27T03:42:48.275 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:17.005 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:48.346 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:17.076 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:48.387 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.386+0000 7ff527947640 10 monclient: get_monmap_and_config 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.386+0000 7ff527947640 10 monclient: build_initial_monmap 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.386+0000 7ff527947640 10 monclient: monmap: 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:42:48.388005+0000 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:42:48.388005+0000 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:48.388 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:48.389 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:48.389 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:48.389 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.387+0000 7ff527947640 1 Processor -- start 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.387+0000 7ff527947640 1 -- start start 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.387+0000 7ff527947640 10 monclient: init 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5200b4c80 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff5200a9e20 0x7ff520142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 10 monclient(hunting): picked mon.noname-a con 0x7ff5200a9e20 addr v2:172.21.15.110:3300/0 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff52013fa10 addr v1:172.21.15.162:6789/0 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5200a99d0 con 0x7ff5200b4c80 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5200b3ed0 con 0x7ff52013fa10 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:48.391 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5200b5060 con 0x7ff5200a9e20 2024-01-27T03:42:48.392 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:48.392 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.388+0000 7ff527947640 10 monclient(hunting): authenticate will time out at 1451.742938s 2024-01-27T03:42:48.392 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.389+0000 7ff5256e3640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff5200b4c80 0x7ff52013f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54880/0 (socket says 172.21.15.110:54880) 2024-01-27T03:42:48.392 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.389+0000 7ff5256e3640 1 -- 172.21.15.110:0/641102485 learned_addr learned my addr 172.21.15.110:0/641102485 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:42:48.395 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.393+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431145427 0 0) 0x7ff5200b3ed0 con 0x7ff52013fa10 2024-01-27T03:42:48.395 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.393+0000 7ff517fff640 10 monclient(hunting): _negotiate 2024-01-27T03:42:48.395 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.393+0000 7ff517fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.393+0000 7ff517fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.393+0000 7ff517fff640 10 monclient(hunting): my global_id is 4201 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.393+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff50c003520 con 0x7ff52013fa10 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.393+0000 7ff524ee2640 1 --2- 172.21.15.110:0/641102485 >> v2:172.21.15.110:3300/0 conn(0x7ff5200a9e20 0x7ff520142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff524ee2640 10 monclient(hunting): get_auth_request con 0x7ff5200a9e20 auth_method 0 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff524ee2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff524ee2640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff524ee2640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239554227 0 0) 0x7ff5200a99d0 con 0x7ff5200b4c80 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff517fff640 10 monclient(hunting): _negotiate 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff517fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff517fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff517fff640 10 monclient(hunting): my global_id is 4200 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.394+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5200b3ed0 con 0x7ff5200b4c80 2024-01-27T03:42:48.396 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.395+0000 7ff524ee2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:48.397 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.395+0000 7ff524ee2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:48.397 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.395+0000 7ff524ee2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:48.408 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 <== mon.1 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1834581503 0 0) 0x7ff50c003520 con 0x7ff52013fa10 2024-01-27T03:42:48.408 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 >> v2:172.21.15.110:3300/0 conn(0x7ff5200a9e20 msgr2=0x7ff520142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:48.408 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 --2- 172.21.15.110:0/641102485 >> v2:172.21.15.110:3300/0 conn(0x7ff5200a9e20 0x7ff520142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.408 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 >> v1:172.21.15.110:6789/0 conn(0x7ff5200b4c80 legacy=0x7ff52013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 10 monclient: _finish_hunting 0 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 monclient: found mon.noname-c-legacy 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.162:6789/0 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5200b3b60 con 0x7ff52013fa10 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 10 monclient: _finish_auth 0 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 10 monclient: _check_auth_tickets 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 <== mon.1 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 218001308 0 0) 0x7ff5180019f0 con 0x7ff52013fa10 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 10 monclient: dump: 2024-01-27T03:42:48.409 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] but i'm connected to v1:172.21.15.162:6789/0, reconnecting 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 >> v1:172.21.15.162:6789/0 conn(0x7ff52013fa10 legacy=0x7ff52014c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.398+0000 7ff517fff640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff50c003b50 0x7ff50c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:48.410 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 10 monclient(hunting): picked mon.a con 0x7ff50c003b50 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c0047c0 0x7ff50c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 10 monclient(hunting): picked mon.b con 0x7ff50c0047c0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff50c005180 0x7ff50c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 10 monclient(hunting): picked mon.c con 0x7ff50c005180 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200b5060 con 0x7ff50c003b50 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c00c050 con 0x7ff50c0047c0 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c00c210 con 0x7ff50c005180 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.399+0000 7ff517fff640 10 monclient(hunting): _finish_auth 0 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.400+0000 7ff525ee4640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff50c005180 0x7ff50c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.400+0000 7ff525ee4640 10 monclient(hunting): get_auth_request con 0x7ff50c005180 auth_method 0 2024-01-27T03:42:48.411 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.400+0000 7ff525ee4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.400+0000 7ff525ee4640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.400+0000 7ff525ee4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.400+0000 7ff525ee4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.400+0000 7ff525ee4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.400+0000 7ff525ee4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff524ee2640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c0047c0 0x7ff50c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 10 monclient(hunting): handle_auth_done global_id 4201 payload 293 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 1 -- 172.21.15.110:0/641102485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c0047c0 msgr2=0x7ff50c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c0047c0 0x7ff50c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 1 -- 172.21.15.110:0/641102485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff50c003b50 msgr2=0x7ff50c003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff50c003b50 0x7ff50c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 10 monclient: _finish_hunting 0 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 1 monclient: found mon.c 2024-01-27T03:42:48.412 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:48.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:48.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 1 -- 172.21.15.110:0/641102485 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c003520 con 0x7ff50c005180 2024-01-27T03:42:48.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 10 monclient: _finish_auth 0 2024-01-27T03:42:48.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 10 monclient: _check_auth_tickets 2024-01-27T03:42:48.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.403+0000 7ff525ee4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:48.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.404+0000 7ff527947640 5 monclient: authenticate success, global_id 4201 2024-01-27T03:42:48.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.404+0000 7ff527947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:42:48.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.404+0000 7ff525ee4640 1 -- 172.21.15.110:0/641102485 reap_dead start 2024-01-27T03:42:48.413 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.406+0000 7ff525ee4640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff50c005180 0x7ff50c00b980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff51800c490 tx=0x7ff50c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1f662ca65c89fb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:48.415 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.412+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51802cc40 con 0x7ff50c005180 2024-01-27T03:42:48.415 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.412+0000 7ff517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:48.415 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.412+0000 7ff517fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:48.415 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff517fff640 10 monclient: dump: 2024-01-27T03:42:48.415 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:48.415 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51802cda0 con 0x7ff50c005180 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff527947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff517fff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff527947640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff527947640 4 set_mon_vals no callback set 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff517fff640 1 -- 172.21.15.110:0/641102485 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518034680 con 0x7ff50c005180 2024-01-27T03:42:48.416 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff517fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.413+0000 7ff517fff640 10 monclient: dump: 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.414+0000 7ff527947640 10 monclient: shutdown 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.414+0000 7ff527947640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:48.417 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.414+0000 7ff527947640 1 -- 172.21.15.110:0/641102485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff50c005180 msgr2=0x7ff50c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:48.418 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.414+0000 7ff527947640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff50c005180 0x7ff50c00b980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff51800c490 tx=0x7ff50c0057f0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.423 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.414+0000 7ff527947640 1 -- 172.21.15.110:0/641102485 shutdown_connections 2024-01-27T03:42:48.423 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.414+0000 7ff527947640 1 --2- 172.21.15.110:0/641102485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff50c005180 0x7ff50c00b980 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.423 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.414+0000 7ff527947640 1 -- 172.21.15.110:0/641102485 >> 172.21.15.110:0/641102485 conn(0x7ff520071f90 msgr2=0x7ff5200723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:48.423 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.415+0000 7ff527947640 1 -- 172.21.15.110:0/641102485 shutdown_connections 2024-01-27T03:42:48.423 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.415+0000 7ff527947640 1 -- 172.21.15.110:0/641102485 wait complete. 2024-01-27T03:42:48.423 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.416+0000 7ff527947640 10 monclient: build_initial_monmap 2024-01-27T03:42:48.423 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.416+0000 7ff527947640 10 monclient: monmap: 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.416+0000 7ff527947640 1 librados: starting msgr at 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.416+0000 7ff527947640 1 librados: starting objecter 2024-01-27T03:42:48.424 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.416+0000 7ff527947640 1 Processor -- start 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.416+0000 7ff527947640 1 -- start start 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.416+0000 7ff527947640 1 librados: setting wanted keys 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.416+0000 7ff527947640 1 librados: calling monclient init 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.416+0000 7ff527947640 10 monclient: init 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.417+0000 7ff527947640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.417+0000 7ff527947640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.417+0000 7ff527947640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c005180 0x7ff52006e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.417+0000 7ff527947640 10 monclient(hunting): picked mon.noname-b con 0x7ff50c005180 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.417+0000 7ff527947640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff520070ff0 0x7ff52006f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.417+0000 7ff527947640 10 monclient(hunting): picked mon.noname-a con 0x7ff520070ff0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.417+0000 7ff527947640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5201dea60 0x7ff5201deee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff527947640 10 monclient(hunting): picked mon.noname-c con 0x7ff5201dea60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff527947640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff527947640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff520070410 con 0x7ff520070ff0 2024-01-27T03:42:48.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff527947640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff527947640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff520070610 con 0x7ff50c005180 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff527947640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff527947640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200708f0 con 0x7ff5201dea60 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff527947640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff527947640 10 monclient(hunting): authenticate will time out at 1451.772045s 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff524ee2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff520070ff0 0x7ff52006f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff525ee4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5201dea60 0x7ff5201deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.418+0000 7ff5256e3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c005180 0x7ff52006e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff524ee2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff520070ff0 0x7ff52006f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:35800/0 (socket says 172.21.15.110:35800) 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff524ee2640 1 -- 172.21.15.110:0/472112471 learned_addr learned my addr 172.21.15.110:0/472112471 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff524ee2640 10 monclient(hunting): get_auth_request con 0x7ff520070ff0 auth_method 0 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff524ee2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff524ee2640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:48.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff524ee2640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff525ee4640 10 monclient(hunting): get_auth_request con 0x7ff5201dea60 auth_method 0 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff525ee4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff525ee4640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff525ee4640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff5256e3640 10 monclient(hunting): get_auth_request con 0x7ff50c005180 auth_method 0 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff5256e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff5256e3640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.419+0000 7ff5256e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.420+0000 7ff5256e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.420+0000 7ff5256e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.420+0000 7ff5256e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.420+0000 7ff525ee4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.420+0000 7ff525ee4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.420+0000 7ff525ee4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:48.427 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.420+0000 7ff524ee2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:48.428 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.420+0000 7ff524ee2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:48.428 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.420+0000 7ff524ee2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:48.429 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 10 monclient(hunting): handle_auth_done global_id 4204 payload 824 2024-01-27T03:42:48.429 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 1 -- 172.21.15.110:0/472112471 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5201dea60 msgr2=0x7ff5201deee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:48.429 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5201dea60 0x7ff5201deee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 1 -- 172.21.15.110:0/472112471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff520070ff0 msgr2=0x7ff52006f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff520070ff0 0x7ff52006f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 10 monclient: _finish_hunting 0 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 1 monclient: found mon.noname-b 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.423+0000 7ff5256e3640 1 -- 172.21.15.110:0/472112471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52006ee90 con 0x7ff50c005180 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.426+0000 7ff5256e3640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c005180 0x7ff52006e950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff520067f30 tx=0x7ff51c00c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.426+0000 7ff515ffb640 10 client.?.objecter ms_handle_connect 0x7ff50c005180 2024-01-27T03:42:48.430 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.426+0000 7ff515ffb640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:48.434 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 1 -- 172.21.15.110:0/472112471 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c005ce0 con 0x7ff50c005180 2024-01-27T03:42:48.434 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:48.434 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:42:48.434 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 10 monclient: dump: 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 10 monclient: _finish_auth 0 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 10 monclient: _check_auth_tickets 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 1 -- 172.21.15.110:0/472112471 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51c005e60 con 0x7ff50c005180 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff527947640 5 monclient: authenticate success, global_id 4204 2024-01-27T03:42:48.435 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff527947640 10 monclient: _renew_subs 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff527947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52006f960 con 0x7ff50c005180 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 1 -- 172.21.15.110:0/472112471 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c0072f0 con 0x7ff50c005180 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff515ffb640 10 monclient: dump: 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:48.436 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff5177fe640 4 set_mon_vals no callback set 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff527947640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff527947640 10 monclient: _renew_subs 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff527947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.431+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff520071430 con 0x7ff50c005180 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.432+0000 7ff527947640 1 librados: init done 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.433+0000 7ff527947640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.433+0000 7ff527947640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.433+0000 7ff527947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:48.437 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.433+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff520071650 con 0x7ff50c005180 2024-01-27T03:42:48.463 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.462+0000 7ff515ffb640 1 -- 172.21.15.110:0/472112471 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 4) v1 ==== 89194+0+0 (secure 0 0 0) 0x7ff51c007450 con 0x7ff50c005180 2024-01-27T03:42:48.463 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.463+0000 7ff515ffb640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff50406c940 0x7ff50406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:48.464 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.463+0000 7ff524ee2640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff50406c940 0x7ff50406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:48.464 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.463+0000 7ff524ee2640 10 monclient: get_auth_request con 0x7ff50406c940 auth_method 0 2024-01-27T03:42:48.464 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.464+0000 7ff524ee2640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff50406c940 0x7ff50406ee00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff52006e500 tx=0x7ff510008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:48.464 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.464+0000 7ff515ffb640 10 client.4204.objecter ms_handle_connect 0x7ff50406c940 2024-01-27T03:42:48.502 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:17.210 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:48.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.580+0000 7ff515ffb640 1 -- 172.21.15.110:0/472112471 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7ff51c057c80 con 0x7ff50c005180 2024-01-27T03:42:48.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.580+0000 7ff515ffb640 10 client.4204.objecter ms_dispatch 0x7ff520000b90 osd_map(10..10 src has 1..10) v4 2024-01-27T03:42:48.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.580+0000 7ff515ffb640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-01-27T03:42:48.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.580+0000 7ff515ffb640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-01-27T03:42:48.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.580+0000 7ff515ffb640 20 client.4204.objecter dump_active .. 0 homeless 2024-01-27T03:42:48.589 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:17.317 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:48.670 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:17.388 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:48.675 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:17.391 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:48.760 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.759+0000 7ff515ffb640 1 -- 172.21.15.110:0/472112471 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7ff51c053f20 con 0x7ff50c005180 2024-01-27T03:42:48.760 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.759+0000 7ff515ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:48.760 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.759+0000 7ff515ffb640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:48.966 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.964+0000 7ff527947640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:48.966 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.964+0000 7ff527947640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:48.966 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.964+0000 7ff527947640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:42:48.966 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.964+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff52004f2a0 con 0x7ff50c005180 2024-01-27T03:42:48.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.981+0000 7ff515ffb640 1 -- 172.21.15.110:0/472112471 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7ff51c017070 con 0x7ff50c005180 2024-01-27T03:42:48.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.981+0000 7ff515ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:48.982 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.981+0000 7ff515ffb640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:48.983 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-27T03:42:48.983 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:42:30.608380+0000","last_up_change":"0.000000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:42:48.986 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 10 librados: watch_flush enter 2024-01-27T03:42:48.986 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 10 librados: watch_flush exit 2024-01-27T03:42:48.986 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 20 client.4204.objecter shutdown clearing up homeless session... 2024-01-27T03:42:48.986 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 10 client.4204.objecter successfully canceled tick 2024-01-27T03:42:48.986 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff50406c940 msgr2=0x7ff50406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:48.986 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff50406c940 0x7ff50406ee00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff52006e500 tx=0x7ff510008040 comp rx=0 tx=0).stop 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 10 monclient: shutdown 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c005180 msgr2=0x7ff52006e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c005180 0x7ff52006e950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff520067f30 tx=0x7ff51c00c890 comp rx=0 tx=0).stop 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 shutdown_connections 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff50406c940 0x7ff50406ee00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5201dea60 0x7ff5201deee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff520070ff0 0x7ff52006f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 --2- 172.21.15.110:0/472112471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff50c005180 0x7ff52006e950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 >> 172.21.15.110:0/472112471 conn(0x7ff520071f90 msgr2=0x7ff5200b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:48.987 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 shutdown_connections 2024-01-27T03:42:48.988 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 -- 172.21.15.110:0/472112471 wait complete. 2024-01-27T03:42:48.988 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:48.984+0000 7ff527947640 1 librados: shutdown 2024-01-27T03:42:48.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-27T03:42:53.645 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:22.367 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:53.648 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:22.371 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:53.986 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:22.716 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:53.990 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:22.720 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.099 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:22.824 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.102 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:22.828 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.147 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:22.856 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.152 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:22.860 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.152 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:22.870 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.155 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:22.884 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.156 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:22.874 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.161 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:22.890 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.267 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:22.995 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.273 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:23.001 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.591 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:23.306 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:54.597 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:23.313 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:55.804 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-27T03:42:55.908 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.907+0000 7fd5a0fd0640 10 monclient: get_monmap_and_config 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.907+0000 7fd5a0fd0640 10 monclient: build_initial_monmap 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.907+0000 7fd5a0fd0640 10 monclient: monmap: 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:42:55.909031+0000 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:42:55.909031+0000 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:42:55.909 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:42:55.910 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.908+0000 7fd5a0fd0640 1 Processor -- start 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.908+0000 7fd5a0fd0640 1 -- start start 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.908+0000 7fd5a0fd0640 10 monclient: init 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd59c0b4c80 0x7fd59c13f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 10 monclient(hunting): picked mon.noname-c con 0x7fd59c0b4c80 addr v2:172.21.15.162:3300/0 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd59c0a9e20 0x7fd59c13f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 10 monclient(hunting): picked mon.noname-a con 0x7fd59c0a9e20 addr v2:172.21.15.110:3300/0 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd59c142060 addr v1:172.21.15.110:6789/0 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c0a99d0 con 0x7fd59c142060 2024-01-27T03:42:55.912 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59c0b5060 con 0x7fd59c0a9e20 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59c0b3ed0 con 0x7fd59c0b4c80 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.909+0000 7fd5a0fd0640 10 monclient(hunting): authenticate will time out at 1459.263436s 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd59a59c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd59c0b4c80 0x7fd59c13f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd599d9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd59c0a9e20 0x7fd59c13f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd59ad9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd59c142060 0x7fd59c142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53370/0 (socket says 172.21.15.110:53370) 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd59ad9d640 1 -- 172.21.15.110:0/2450539965 learned_addr learned my addr 172.21.15.110:0/2450539965 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd59a59c640 10 monclient(hunting): get_auth_request con 0x7fd59c0b4c80 auth_method 0 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd59a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd59a59c640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd59a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd599d9b640 10 monclient(hunting): get_auth_request con 0x7fd59c0a9e20 auth_method 0 2024-01-27T03:42:55.913 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd599d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:55.914 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd599d9b640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:55.914 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.910+0000 7fd599d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:55.914 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.911+0000 7fd59a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:55.914 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.911+0000 7fd59a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:55.914 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.911+0000 7fd59a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:55.914 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.911+0000 7fd599d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:55.914 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.911+0000 7fd599d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:55.914 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.911+0000 7fd599d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:55.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.913+0000 7fd599d9b640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2024-01-27T03:42:55.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.913+0000 7fd599d9b640 1 -- 172.21.15.110:0/2450539965 >> v2:172.21.15.162:3300/0 conn(0x7fd59c0b4c80 msgr2=0x7fd59c13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:42:55.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.913+0000 7fd599d9b640 1 --2- 172.21.15.110:0/2450539965 >> v2:172.21.15.162:3300/0 conn(0x7fd59c0b4c80 0x7fd59c13f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:55.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.913+0000 7fd599d9b640 1 -- 172.21.15.110:0/2450539965 >> v1:172.21.15.110:6789/0 conn(0x7fd59c142060 legacy=0x7fd59c142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:55.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.913+0000 7fd599d9b640 10 monclient: _finish_hunting 0 2024-01-27T03:42:55.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.913+0000 7fd599d9b640 1 monclient: found mon.noname-a 2024-01-27T03:42:55.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.913+0000 7fd599d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:55.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.913+0000 7fd599d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:42:55.915 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.913+0000 7fd599d9b640 1 -- 172.21.15.110:0/2450539965 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c0b3b60 con 0x7fd59c0a9e20 2024-01-27T03:42:55.918 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.918+0000 7fd599d9b640 1 --2- 172.21.15.110:0/2450539965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c0a9e20 0x7fd59c13f840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd59c066fa0 tx=0x7fd59002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8a11c027006b23 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 1 -- 172.21.15.110:0/2450539965 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590031640 con 0x7fd59c0a9e20 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 10 monclient: dump: 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:55.925 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 10 monclient: _finish_auth 0 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 10 monclient: _check_auth_tickets 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 1 -- 172.21.15.110:0/2450539965 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5900317a0 con 0x7fd59c0a9e20 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 1 -- 172.21.15.110:0/2450539965 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590031b70 con 0x7fd59c0a9e20 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.922+0000 7fd59959a640 10 monclient: dump: 2024-01-27T03:42:55.926 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 5 monclient: authenticate success, global_id 4209 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 10 monclient: get_monmap_and_config success 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 4 set_mon_vals no callback set 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 10 monclient: shutdown 2024-01-27T03:42:55.927 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:55.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2450539965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c0a9e20 msgr2=0x7fd59c13f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:55.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 1 --2- 172.21.15.110:0/2450539965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c0a9e20 0x7fd59c13f840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd59c066fa0 tx=0x7fd59002b8f0 comp rx=0 tx=0).stop 2024-01-27T03:42:55.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2450539965 shutdown_connections 2024-01-27T03:42:55.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 1 --2- 172.21.15.110:0/2450539965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c0a9e20 0x7fd59c13f840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:55.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 1 --2- 172.21.15.110:0/2450539965 >> v2:172.21.15.162:3300/0 conn(0x7fd59c0b4c80 0x7fd59c13f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:55.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2450539965 >> 172.21.15.110:0/2450539965 conn(0x7fd59c071f90 msgr2=0x7fd59c0723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:55.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.923+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2450539965 shutdown_connections 2024-01-27T03:42:55.928 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.924+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2450539965 wait complete. 2024-01-27T03:42:55.934 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 10 monclient: build_initial_monmap 2024-01-27T03:42:55.934 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 10 monclient: monmap: 2024-01-27T03:42:55.934 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:42:55.934 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 1 librados: starting msgr at 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 1 librados: starting objecter 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 1 Processor -- start 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 1 -- start start 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 1 librados: setting wanted keys 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 1 librados: calling monclient init 2024-01-27T03:42:55.935 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 10 monclient: init 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 10 monclient: _reopen_session rank -1 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd59c0b4c80 0x7fd59c1374e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 10 monclient(hunting): picked mon.noname-b con 0x7fd59c0b4c80 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd59c142060 0x7fd59c070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 10 monclient(hunting): picked mon.noname-c con 0x7fd59c142060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.925+0000 7fd5a0fd0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c06fc60 0x7fd59c1db4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd5a0fd0640 10 monclient(hunting): picked mon.noname-a con 0x7fd59c06fc60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd5a0fd0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd5a0fd0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0707f0 con 0x7fd59c06fc60 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd5a0fd0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd5a0fd0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c1381f0 con 0x7fd59c0b4c80 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd5a0fd0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:42:55.936 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd5a0fd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c070c80 con 0x7fd59c142060 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd5a0fd0640 10 monclient(hunting): _renew_subs 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd5a0fd0640 10 monclient(hunting): authenticate will time out at 1459.280335s 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd59ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c06fc60 0x7fd59c1db4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd599d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd59c142060 0x7fd59c070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd59a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd59c0b4c80 0x7fd59c1374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd59ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c06fc60 0x7fd59c1db4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45926/0 (socket says 172.21.15.110:45926) 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd59ad9d640 1 -- 172.21.15.110:0/2635297322 learned_addr learned my addr 172.21.15.110:0/2635297322 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd59ad9d640 10 monclient(hunting): get_auth_request con 0x7fd59c06fc60 auth_method 0 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd59ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd59ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.926+0000 7fd59ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd599d9b640 10 monclient(hunting): get_auth_request con 0x7fd59c142060 auth_method 0 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd599d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:55.937 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd599d9b640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd599d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59a59c640 10 monclient(hunting): get_auth_request con 0x7fd59c0b4c80 auth_method 0 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59a59c640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd599d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd599d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd599d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.927+0000 7fd59a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 10 monclient(hunting): handle_auth_done global_id 4190 payload 824 2024-01-27T03:42:55.938 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 1 -- 172.21.15.110:0/2635297322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd59c0b4c80 msgr2=0x7fd59c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd59c0b4c80 0x7fd59c1374e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 1 -- 172.21.15.110:0/2635297322 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c06fc60 msgr2=0x7fd59c1db4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c06fc60 0x7fd59c1db4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 10 monclient: _finish_hunting 0 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 1 monclient: found mon.noname-c 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.930+0000 7fd599d9b640 1 -- 172.21.15.110:0/2635297322 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd590009400 con 0x7fd59c142060 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.932+0000 7fd599d9b640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd59c142060 0x7fd59c070140 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd590033000 tx=0x7fd5900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.932+0000 7fd58affd640 10 client.?.objecter ms_handle_connect 0x7fd59c142060 2024-01-27T03:42:55.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.932+0000 7fd58affd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:42:55.940 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.936+0000 7fd58affd640 1 -- 172.21.15.110:0/2635297322 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59003ac60 con 0x7fd59c142060 2024-01-27T03:42:55.940 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:55.940 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:42:55.940 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 10 monclient: dump: 2024-01-27T03:42:55.940 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 10 monclient: _finish_auth 0 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 10 monclient: _check_auth_tickets 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 1 -- 172.21.15.110:0/2635297322 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59003adc0 con 0x7fd59c142060 2024-01-27T03:42:55.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd5a0fd0640 5 monclient: authenticate success, global_id 4190 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd5a0fd0640 10 monclient: _renew_subs 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd5a0fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c1db9f0 con 0x7fd59c142060 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 1 -- 172.21.15.110:0/2635297322 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590044640 con 0x7fd59c142060 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd58affd640 10 monclient: dump: 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:42:55.942 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd5a0fd0640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd5a0fd0640 10 monclient: _renew_subs 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd5a0fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c1dbdb0 con 0x7fd59c142060 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd598d99640 4 set_mon_vals no callback set 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.937+0000 7fd5a0fd0640 1 librados: init done 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.938+0000 7fd5a0fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.938+0000 7fd5a0fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.938+0000 7fd5a0fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:55.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.938+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c070960 con 0x7fd59c142060 2024-01-27T03:42:55.962 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.961+0000 7fd58affd640 1 -- 172.21.15.110:0/2635297322 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 4) v1 ==== 89194+0+0 (secure 0 0 0) 0x7fd59002f050 con 0x7fd59c142060 2024-01-27T03:42:55.963 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.962+0000 7fd58affd640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd57c06c960 0x7fd57c06ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:42:55.963 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.962+0000 7fd59a59c640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd57c06c960 0x7fd57c06ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:42:55.963 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.962+0000 7fd59a59c640 10 monclient: get_auth_request con 0x7fd57c06c960 auth_method 0 2024-01-27T03:42:55.964 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.963+0000 7fd58affd640 10 client.4190.objecter ms_handle_connect 0x7fd57c06c960 2024-01-27T03:42:55.964 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.963+0000 7fd59a59c640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd57c06c960 0x7fd57c06ee20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009b40 tx=0x7fd58c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:42:55.965 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.964+0000 7fd58affd640 1 -- 172.21.15.110:0/2635297322 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd59007a530 con 0x7fd59c142060 2024-01-27T03:42:55.965 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.964+0000 7fd58affd640 10 client.4190.objecter ms_dispatch 0x7fd59c000b90 osd_map(10..10 src has 1..10) v4 2024-01-27T03:42:55.965 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.964+0000 7fd58affd640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-01-27T03:42:55.965 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.964+0000 7fd58affd640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-01-27T03:42:55.965 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:55.964+0000 7fd58affd640 20 client.4190.objecter dump_active .. 0 homeless 2024-01-27T03:42:56.354 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.353+0000 7fd58affd640 1 -- 172.21.15.110:0/2635297322 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fd5900767d0 con 0x7fd59c142060 2024-01-27T03:42:56.354 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.353+0000 7fd58affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:42:56.355 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.353+0000 7fd58affd640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:42:56.562 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.561+0000 7fd5a0fd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:56.562 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.561+0000 7fd5a0fd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:56.562 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.561+0000 7fd5a0fd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:42:56.562 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.561+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd59c14c8a0 con 0x7fd59c142060 2024-01-27T03:42:56.635 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.634+0000 7fd58affd640 1 -- 172.21.15.110:0/2635297322 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fd590035090 con 0x7fd59c142060 2024-01-27T03:42:56.635 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.634+0000 7fd58affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:42:56.635 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.634+0000 7fd58affd640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:42:56.636 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-27T03:42:56.636 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:42:30.608380+0000","last_up_change":"0.000000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:42:56.638 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 10 librados: watch_flush enter 2024-01-27T03:42:56.638 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 10 librados: watch_flush exit 2024-01-27T03:42:56.638 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 20 client.4190.objecter shutdown clearing up homeless session... 2024-01-27T03:42:56.638 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 10 client.4190.objecter successfully canceled tick 2024-01-27T03:42:56.638 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd57c06c960 msgr2=0x7fd57c06ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:56.638 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd57c06c960 0x7fd57c06ee20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009b40 tx=0x7fd58c009190 comp rx=0 tx=0).stop 2024-01-27T03:42:56.639 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 10 monclient: shutdown 2024-01-27T03:42:56.639 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:42:56.639 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd59c142060 msgr2=0x7fd59c070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:42:56.639 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.637+0000 7fd5a0fd0640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd59c142060 0x7fd59c070140 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd590033000 tx=0x7fd5900046c0 comp rx=0 tx=0).stop 2024-01-27T03:42:56.639 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.638+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 shutdown_connections 2024-01-27T03:42:56.639 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.638+0000 7fd5a0fd0640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd57c06c960 0x7fd57c06ee20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:56.639 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.638+0000 7fd5a0fd0640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd59c142060 0x7fd59c070140 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:56.639 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.638+0000 7fd5a0fd0640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd59c06fc60 0x7fd59c1db4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:56.639 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.638+0000 7fd5a0fd0640 1 --2- 172.21.15.110:0/2635297322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd59c0b4c80 0x7fd59c1374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:42:56.640 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.638+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 >> 172.21.15.110:0/2635297322 conn(0x7fd59c071f90 msgr2=0x7fd59c0b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:42:56.640 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.638+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 shutdown_connections 2024-01-27T03:42:56.640 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.639+0000 7fd5a0fd0640 1 -- 172.21.15.110:0/2635297322 wait complete. 2024-01-27T03:42:56.640 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:42:56.639+0000 7fd5a0fd0640 1 librados: shutdown 2024-01-27T03:42:56.653 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-27T03:42:57.871 INFO:tasks.ceph.osd.7.smithi162.stderr:==00:00:00:26.594 34673== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:58.206 INFO:tasks.ceph.osd.4.smithi162.stderr:==00:00:00:26.936 34670== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:58.441 INFO:tasks.ceph.osd.6.smithi162.stderr:==00:00:00:27.167 34672== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:58.452 INFO:tasks.ceph.osd.5.smithi162.stderr:==00:00:00:27.182 34671== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:58.454 INFO:tasks.ceph.osd.2.smithi110.stderr:==00:00:00:27.162 34883== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:58.746 INFO:tasks.ceph.osd.3.smithi110.stderr:==00:00:00:27.464 34881== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:58.859 INFO:tasks.ceph.osd.0.smithi110.stderr:==00:00:00:27.588 34880== Warning: unimplemented fcntl command: 1038 2024-01-27T03:42:59.451 INFO:tasks.ceph.osd.1.smithi110.stderr:==00:00:00:28.167 34882== Warning: unimplemented fcntl command: 1038 2024-01-27T03:43:03.460 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-27T03:43:03.550 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.548+0000 7f60397c0640 10 monclient: get_monmap_and_config 2024-01-27T03:43:03.550 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.548+0000 7f60397c0640 10 monclient: build_initial_monmap 2024-01-27T03:43:03.550 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.549+0000 7f60397c0640 10 monclient: monmap: 2024-01-27T03:43:03.550 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:43:03.550 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:03.550 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:43:03.550129+0000 2024-01-27T03:43:03.550 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:43:03.550129+0000 2024-01-27T03:43:03.550 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.549+0000 7f60397c0640 1 Processor -- start 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.549+0000 7f60397c0640 1 -- start start 2024-01-27T03:43:03.551 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.549+0000 7f60397c0640 10 monclient: init 2024-01-27T03:43:03.553 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:03.553 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-01-27T03:43:03.553 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60340b4c80 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f60340a9e20 0x7f6034142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 10 monclient(hunting): picked mon.noname-a con 0x7f60340a9e20 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f603413fa10 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60340a99d0 con 0x7f603413fa10 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60340b3ed0 con 0x7f60340b4c80 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60340b5060 con 0x7f60340a9e20 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.550+0000 7f60397c0640 10 monclient(hunting): authenticate will time out at 1466.904231s 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.551+0000 7f60327fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f60340a9e20 0x7f6034142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.551+0000 7f6032ffd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f60340b4c80 0x7f603413f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:42018/0 (socket says 172.21.15.110:42018) 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.551+0000 7f6032ffd640 1 -- 172.21.15.110:0/545134534 learned_addr learned my addr 172.21.15.110:0/545134534 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:03.554 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.551+0000 7f60327fc640 10 monclient(hunting): get_auth_request con 0x7f60340a9e20 auth_method 0 2024-01-27T03:43:03.555 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.551+0000 7f60327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:03.555 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.551+0000 7f60327fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:03.555 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.551+0000 7f60327fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:03.555 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.552+0000 7f60327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:03.555 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.552+0000 7f60327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:03.555 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.552+0000 7f60327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:03.557 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.555+0000 7f60327fc640 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2024-01-27T03:43:03.557 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.555+0000 7f6031ffb640 1 -- 172.21.15.110:0/545134534 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106074379 0 0) 0x7f60340b3ed0 con 0x7f60340b4c80 2024-01-27T03:43:03.557 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.556+0000 7f60327fc640 1 -- 172.21.15.110:0/545134534 >> v1:172.21.15.162:6789/0 conn(0x7f60340b4c80 legacy=0x7f603413f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:03.557 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.556+0000 7f60327fc640 1 -- 172.21.15.110:0/545134534 >> v1:172.21.15.110:6789/0 conn(0x7f603413fa10 legacy=0x7f603414c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:03.557 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.556+0000 7f60327fc640 10 monclient: _finish_hunting 0 2024-01-27T03:43:03.557 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.556+0000 7f60327fc640 1 monclient: found mon.noname-a 2024-01-27T03:43:03.557 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.556+0000 7f60327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:03.558 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.556+0000 7f60327fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:03.558 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.556+0000 7f60327fc640 1 -- 172.21.15.110:0/545134534 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60340b3b60 con 0x7f60340a9e20 2024-01-27T03:43:03.558 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.556+0000 7f6031ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-27T03:43:03.562 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.561+0000 7f60327fc640 1 --2- 172.21.15.110:0/545134534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60340a9e20 0x7f6034142060 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60280097a0 tx=0x7f602802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1ac079c43b2655 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 1 -- 172.21.15.110:0/545134534 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028030610 con 0x7f60340a9e20 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 10 monclient: dump: 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:03.574 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 10 monclient: _finish_auth 0 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 10 monclient: _check_auth_tickets 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f60397c0640 5 monclient: authenticate success, global_id 4218 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f60397c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 1 -- 172.21.15.110:0/545134534 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6028030770 con 0x7f60340a9e20 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f60397c0640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f60397c0640 4 set_mon_vals no callback set 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 1 -- 172.21.15.110:0/545134534 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028030610 con 0x7f60340a9e20 2024-01-27T03:43:03.575 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.566+0000 7f6031ffb640 10 monclient: dump: 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:03.576 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.567+0000 7f60397c0640 10 monclient: shutdown 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.567+0000 7f60397c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.567+0000 7f60397c0640 1 -- 172.21.15.110:0/545134534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60340a9e20 msgr2=0x7f6034142060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.567+0000 7f60397c0640 1 --2- 172.21.15.110:0/545134534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60340a9e20 0x7f6034142060 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60280097a0 tx=0x7f602802b960 comp rx=0 tx=0).stop 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.567+0000 7f60397c0640 1 -- 172.21.15.110:0/545134534 shutdown_connections 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.567+0000 7f60397c0640 1 --2- 172.21.15.110:0/545134534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60340a9e20 0x7f6034142060 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.567+0000 7f60397c0640 1 -- 172.21.15.110:0/545134534 >> 172.21.15.110:0/545134534 conn(0x7f6034071f90 msgr2=0x7f60340723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.567+0000 7f60397c0640 1 -- 172.21.15.110:0/545134534 shutdown_connections 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.567+0000 7f60397c0640 1 -- 172.21.15.110:0/545134534 wait complete. 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.568+0000 7f60397c0640 10 monclient: build_initial_monmap 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.568+0000 7f60397c0640 10 monclient: monmap: 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-27T03:43:03.577 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.568+0000 7f60397c0640 1 librados: starting msgr at 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.568+0000 7f60397c0640 1 librados: starting objecter 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.568+0000 7f60397c0640 1 Processor -- start 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.568+0000 7f60397c0640 1 -- start start 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.568+0000 7f60397c0640 1 librados: setting wanted keys 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.568+0000 7f60397c0640 1 librados: calling monclient init 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.568+0000 7f60397c0640 10 monclient: init 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T03:43:03.578 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60340b4c80 0x7f60341391b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient(hunting): picked mon.noname-c con 0x7f60340b4c80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f603413fa10 0x7f6034071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient(hunting): picked mon.noname-b con 0x7f603413fa10 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60341384d0 0x7f6034138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient(hunting): picked mon.noname-a con 0x7f60341384d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60340705e0 con 0x7f60341384d0 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6034071590 con 0x7f603413fa10 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60340708c0 con 0x7f60340b4c80 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.569+0000 7f60397c0640 10 monclient(hunting): authenticate will time out at 1466.923439s 2024-01-27T03:43:03.579 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f6032ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60340b4c80 0x7f60341391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f60327fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f603413fa10 0x7f6034071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f60327fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f603413fa10 0x7f6034071050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:47376/0 (socket says 172.21.15.110:47376) 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f60327fc640 1 -- 172.21.15.110:0/1002545391 learned_addr learned my addr 172.21.15.110:0/1002545391 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f60327fc640 10 monclient(hunting): get_auth_request con 0x7f603413fa10 auth_method 0 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f60327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f60327fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f60327fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f6032ffd640 10 monclient(hunting): get_auth_request con 0x7f60340b4c80 auth_method 0 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f6032ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f6032ffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f6032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.570+0000 7f60337fe640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60341384d0 0x7f6034138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60337fe640 10 monclient(hunting): get_auth_request con 0x7f60341384d0 auth_method 0 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:03.580 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60337fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60337fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f6032ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f6032ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f6032ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:03.581 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.571+0000 7f60337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:03.583 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2024-01-27T03:43:03.583 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 1 -- 172.21.15.110:0/1002545391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f603413fa10 msgr2=0x7f6034071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f603413fa10 0x7f6034071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 1 -- 172.21.15.110:0/1002545391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60341384d0 msgr2=0x7f6034138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60341384d0 0x7f6034138960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 10 monclient: _finish_hunting 0 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 1 monclient: found mon.noname-c 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.575+0000 7f6032ffd640 1 -- 172.21.15.110:0/1002545391 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6028009400 con 0x7f60340b4c80 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.579+0000 7f6032ffd640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60340b4c80 0x7f60341391b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f602400c3e0 tx=0x7f602400c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.579+0000 7f601b7fe640 10 client.?.objecter ms_handle_connect 0x7f60340b4c80 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.579+0000 7f601b7fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f601b7fe640 1 -- 172.21.15.110:0/1002545391 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60240074e0 con 0x7f60340b4c80 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f601b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:03.584 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f601b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f601b7fe640 10 monclient: dump: 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:03.585 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f601b7fe640 10 monclient: _finish_auth 0 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f601b7fe640 10 monclient: _check_auth_tickets 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f601b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f60397c0640 5 monclient: authenticate success, global_id 4193 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f60397c0640 10 monclient: _renew_subs 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f60397c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6034071850 con 0x7f60340b4c80 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f60397c0640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f60397c0640 10 monclient: _renew_subs 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f60397c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603406e3f0 con 0x7f60340b4c80 2024-01-27T03:43:03.586 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.582+0000 7f60397c0640 1 librados: init done 2024-01-27T03:43:03.587 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.583+0000 7f60397c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:03.587 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.583+0000 7f60397c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:03.587 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.583+0000 7f60397c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:03.588 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.583+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603404f2a0 con 0x7f60340b4c80 2024-01-27T03:43:03.802 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.801+0000 7f601b7fe640 1 -- 172.21.15.110:0/1002545391 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6024007e40 con 0x7f60340b4c80 2024-01-27T03:43:03.802 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.801+0000 7f601b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:03.802 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.801+0000 7f60317fa640 4 set_mon_vals no callback set 2024-01-27T03:43:03.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.978+0000 7f601b7fe640 1 -- 172.21.15.110:0/1002545391 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60240049e0 con 0x7f60340b4c80 2024-01-27T03:43:03.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.978+0000 7f601b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:03.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.978+0000 7f601b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:03.979 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.978+0000 7f601b7fe640 10 monclient: dump: 2024-01-27T03:43:03.979 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:03.979 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:03.979 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:03.980 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:03.980 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:03.980 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:03.980 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:03.980 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:03.980 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:03.980 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:03.981 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.979+0000 7f601b7fe640 1 -- 172.21.15.110:0/1002545391 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 4) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f60240133c0 con 0x7f60340b4c80 2024-01-27T03:43:03.981 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.979+0000 7f601b7fe640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f602006c910 0x7f602006edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:03.981 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.980+0000 7f60327fc640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f602006c910 0x7f602006edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:03.981 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.980+0000 7f60327fc640 10 monclient: get_auth_request con 0x7f602006c910 auth_method 0 2024-01-27T03:43:03.981 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.980+0000 7f60327fc640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f602006c910 0x7f602006edd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f60340a9d50 tx=0x7f6028008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:03.981 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.980+0000 7f601b7fe640 10 client.4193.objecter ms_handle_connect 0x7f602006c910 2024-01-27T03:43:03.993 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.991+0000 7f601b7fe640 1 -- 172.21.15.110:0/1002545391 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f60240586c0 con 0x7f60340b4c80 2024-01-27T03:43:03.993 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.991+0000 7f601b7fe640 10 client.4193.objecter ms_dispatch 0x7f6034000b90 osd_map(10..10 src has 1..10) v4 2024-01-27T03:43:03.993 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.991+0000 7f601b7fe640 3 client.4193.objecter handle_osd_map got epochs [10,10] > 0 2024-01-27T03:43:03.993 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.991+0000 7f601b7fe640 3 client.4193.objecter handle_osd_map decoding full epoch 10 2024-01-27T03:43:03.993 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.992+0000 7f601b7fe640 20 client.4193.objecter dump_active .. 0 homeless 2024-01-27T03:43:03.993 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.992+0000 7f601b7fe640 1 -- 172.21.15.110:0/1002545391 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f6024058490 con 0x7f60340b4c80 2024-01-27T03:43:03.993 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.992+0000 7f601b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:03.993 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:03.992+0000 7f601b7fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:04.271 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.270+0000 7f60397c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:04.271 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.270+0000 7f60397c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:04.271 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.270+0000 7f60397c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:04.272 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.270+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6034071b90 con 0x7f60340b4c80 2024-01-27T03:43:04.288 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.287+0000 7f601b7fe640 1 -- 172.21.15.110:0/1002545391 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f6024058020 con 0x7f60340b4c80 2024-01-27T03:43:04.288 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.287+0000 7f601b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:04.288 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.287+0000 7f601b7fe640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:04.288 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-27T03:43:04.288 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:42:30.608380+0000","last_up_change":"0.000000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:43:04.291 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.289+0000 7f60397c0640 10 librados: watch_flush enter 2024-01-27T03:43:04.291 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.289+0000 7f60397c0640 10 librados: watch_flush exit 2024-01-27T03:43:04.291 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.289+0000 7f60397c0640 20 client.4193.objecter shutdown clearing up homeless session... 2024-01-27T03:43:04.291 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.289+0000 7f60397c0640 10 client.4193.objecter successfully canceled tick 2024-01-27T03:43:04.291 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f602006c910 msgr2=0x7f602006edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:04.291 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f602006c910 0x7f602006edd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f60340a9d50 tx=0x7f6028008030 comp rx=0 tx=0).stop 2024-01-27T03:43:04.291 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 10 monclient: shutdown 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60340b4c80 msgr2=0x7f60341391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60340b4c80 0x7f60341391b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f602400c3e0 tx=0x7f602400c8a0 comp rx=0 tx=0).stop 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 shutdown_connections 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f602006c910 0x7f602006edd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60341384d0 0x7f6034138960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f603413fa10 0x7f6034071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 --2- 172.21.15.110:0/1002545391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60340b4c80 0x7f60341391b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 >> 172.21.15.110:0/1002545391 conn(0x7f6034071f90 msgr2=0x7f60340b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 shutdown_connections 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 -- 172.21.15.110:0/1002545391 wait complete. 2024-01-27T03:43:04.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:04.290+0000 7f60397c0640 1 librados: shutdown 2024-01-27T03:43:04.302 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-27T03:43:05.979 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T03:43:05.974+0000 5dc2480 -1 osd.7 0 log_to_monitors true 2024-01-27T03:43:06.385 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:06.377+0000 5dc2480 -1 osd.4 0 log_to_monitors true 2024-01-27T03:43:06.769 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:06.764+0000 5dc2480 -1 osd.6 0 log_to_monitors true 2024-01-27T03:43:07.053 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:43:07.049+0000 5dc2480 -1 osd.2 0 log_to_monitors true 2024-01-27T03:43:07.137 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:43:07.132+0000 5dc2480 -1 osd.5 0 log_to_monitors true 2024-01-27T03:43:07.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:43:07.308+0000 5dc2480 -1 osd.0 0 log_to_monitors true 2024-01-27T03:43:07.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-27T03:43:07.811+0000 5dc2480 -1 osd.3 0 log_to_monitors true 2024-01-27T03:43:08.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:08.189+0000 5dc2480 -1 osd.1 0 log_to_monitors true 2024-01-27T03:43:08.817 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T03:43:08.812+0000 c494640 -1 osd.7 0 waiting for initial osdmap 2024-01-27T03:43:09.207 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T03:43:09.193+0000 10253640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-27T03:43:10.059 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:43:10.055+0000 c494640 -1 osd.5 0 waiting for initial osdmap 2024-01-27T03:43:10.501 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:43:10.487+0000 10253640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-27T03:43:11.013 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:43:11.007+0000 c494640 -1 osd.0 0 waiting for initial osdmap 2024-01-27T03:43:11.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-27T03:43:11.032+0000 ac7e640 -1 osd.3 0 waiting for initial osdmap 2024-01-27T03:43:11.110 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-27T03:43:11.256 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.252+0000 7f3bf2264640 10 monclient: get_monmap_and_config 2024-01-27T03:43:11.256 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.252+0000 7f3bf2264640 10 monclient: build_initial_monmap 2024-01-27T03:43:11.256 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.252+0000 7f3bf2264640 10 monclient: monmap: 2024-01-27T03:43:11.256 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:43:11.254099+0000 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:43:11.254099+0000 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:11.257 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:11.262 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.254+0000 7f3bf2264640 1 Processor -- start 2024-01-27T03:43:11.262 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.254+0000 7f3bf2264640 1 -- start start 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.254+0000 7f3bf2264640 10 monclient: init 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.254+0000 7f3bf2264640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3bec0b4c80 0x7f3bec13f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 10 monclient(hunting): picked mon.noname-c con 0x7f3bec0b4c80 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0a9e20 0x7f3bec13f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 10 monclient(hunting): picked mon.noname-b con 0x7f3bec0a9e20 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3bec142060 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bec0a99d0 con 0x7f3bec142060 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:11.263 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bec0b5060 con 0x7f3bec0b4c80 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec0b3ed0 con 0x7f3bec0a9e20 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.255+0000 7f3bf2264640 10 monclient(hunting): authenticate will time out at 1474.609134s 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3bec0b4c80 0x7f3bec13f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3bec0b4c80 0x7f3bec13f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:57452/0 (socket says 172.21.15.110:57452) 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 1 -- 172.21.15.110:0/1668196925 learned_addr learned my addr 172.21.15.110:0/1668196925 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 10 monclient(hunting): get_auth_request con 0x7f3bec0b4c80 auth_method 0 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3beb7fe640 1 --2- 172.21.15.110:0/1668196925 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0a9e20 0x7f3bec13f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3beb7fe640 10 monclient(hunting): get_auth_request con 0x7f3bec0a9e20 auth_method 0 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3beb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:11.264 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3beb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3beb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.256+0000 7f3bebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.257+0000 7f3beb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.257+0000 7f3beb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.257+0000 7f3beb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.258+0000 7f3bebfff640 10 monclient(hunting): handle_auth_done global_id 4240 payload 274 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.258+0000 7f3bebfff640 1 -- 172.21.15.110:0/1668196925 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0a9e20 msgr2=0x7f3bec13f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.258+0000 7f3bebfff640 1 --2- 172.21.15.110:0/1668196925 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0a9e20 0x7f3bec13f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.258+0000 7f3bebfff640 1 -- 172.21.15.110:0/1668196925 >> v1:172.21.15.110:6789/0 conn(0x7f3bec142060 legacy=0x7f3bec142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.258+0000 7f3bebfff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:11.265 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.258+0000 7f3bebfff640 1 monclient: found mon.noname-c 2024-01-27T03:43:11.266 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.258+0000 7f3bebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:11.266 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.258+0000 7f3bebfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:11.266 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.258+0000 7f3bebfff640 1 -- 172.21.15.110:0/1668196925 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec0b3b60 con 0x7f3bec0b4c80 2024-01-27T03:43:11.266 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.260+0000 7f3bebfff640 1 --2- 172.21.15.110:0/1668196925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec0b4c80 0x7f3bec13f300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009870 tx=0x7f3bdc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c8078b97aec3e32 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:11.267 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 1 -- 172.21.15.110:0/1668196925 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc030840 con 0x7f3bec0b4c80 2024-01-27T03:43:11.267 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:11.267 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:11.267 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 10 monclient: dump: 2024-01-27T03:43:11.267 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:11.267 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:11.267 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:11.267 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 10 monclient: _finish_auth 0 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 1 -- 172.21.15.110:0/1668196925 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc0309a0 con 0x7f3bec0b4c80 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 1 -- 172.21.15.110:0/1668196925 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc030d30 con 0x7f3bec0b4c80 2024-01-27T03:43:11.268 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3beaffd640 10 monclient: dump: 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3bf2264640 5 monclient: authenticate success, global_id 4240 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3bf2264640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:11.269 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.264+0000 7f3bf2264640 4 set_mon_vals no callback set 2024-01-27T03:43:11.270 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 10 monclient: shutdown 2024-01-27T03:43:11.270 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:11.270 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 1 -- 172.21.15.110:0/1668196925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec0b4c80 msgr2=0x7f3bec13f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:11.270 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 1 --2- 172.21.15.110:0/1668196925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec0b4c80 0x7f3bec13f300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009870 tx=0x7f3bdc02b980 comp rx=0 tx=0).stop 2024-01-27T03:43:11.270 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 1 -- 172.21.15.110:0/1668196925 shutdown_connections 2024-01-27T03:43:11.270 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 1 --2- 172.21.15.110:0/1668196925 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0a9e20 0x7f3bec13f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:11.270 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 1 --2- 172.21.15.110:0/1668196925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec0b4c80 0x7f3bec13f300 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:11.270 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 1 -- 172.21.15.110:0/1668196925 >> 172.21.15.110:0/1668196925 conn(0x7f3bec071f90 msgr2=0x7f3bec0723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:11.270 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 1 -- 172.21.15.110:0/1668196925 shutdown_connections 2024-01-27T03:43:11.276 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.265+0000 7f3bf2264640 1 -- 172.21.15.110:0/1668196925 wait complete. 2024-01-27T03:43:11.276 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.266+0000 7f3bf2264640 10 monclient: build_initial_monmap 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.266+0000 7f3bf2264640 10 monclient: monmap: 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.266+0000 7f3bf2264640 1 librados: starting msgr at 2024-01-27T03:43:11.277 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.266+0000 7f3bf2264640 1 librados: starting objecter 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.266+0000 7f3bf2264640 1 Processor -- start 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.267+0000 7f3bf2264640 1 -- start start 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.267+0000 7f3bf2264640 1 librados: setting wanted keys 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.267+0000 7f3bf2264640 1 librados: calling monclient init 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.267+0000 7f3bf2264640 10 monclient: init 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.267+0000 7f3bf2264640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.267+0000 7f3bf2264640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.267+0000 7f3bf2264640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0b4c80 0x7f3bec13a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.267+0000 7f3bf2264640 10 monclient(hunting): picked mon.noname-c con 0x7f3bec0b4c80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.267+0000 7f3bf2264640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bec142060 0x7f3bec1365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 10 monclient(hunting): picked mon.noname-a con 0x7f3bec142060 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec137180 0x7f3bec1375d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 10 monclient(hunting): picked mon.noname-b con 0x7f3bec137180 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:11.278 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec0ac080 con 0x7f3bec142060 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec137b10 con 0x7f3bec137180 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec0ac300 con 0x7f3bec0b4c80 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf2264640 10 monclient(hunting): authenticate will time out at 1474.622078s 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf0801640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec137180 0x7f3bec1375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf0801640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec137180 0x7f3bec1375d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:57456/0 (socket says 172.21.15.110:57456) 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf0801640 1 -- 172.21.15.110:0/2622855084 learned_addr learned my addr 172.21.15.110:0/2622855084 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf0801640 10 monclient(hunting): get_auth_request con 0x7f3bec137180 auth_method 0 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf0801640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf0801640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bf0801640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:11.279 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3bebfff640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0b4c80 0x7f3bec13a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.268+0000 7f3beb7fe640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bec142060 0x7f3bec1365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3bf0801640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3bf0801640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3bf0801640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3bebfff640 10 monclient(hunting): get_auth_request con 0x7f3bec0b4c80 auth_method 0 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3bebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3bebfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3beb7fe640 10 monclient(hunting): get_auth_request con 0x7f3bec142060 auth_method 0 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3beb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3beb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.269+0000 7f3beb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:11.280 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.270+0000 7f3beb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.270+0000 7f3beb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.270+0000 7f3beb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.270+0000 7f3bebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.270+0000 7f3bebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.270+0000 7f3bebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 10 monclient(hunting): handle_auth_done global_id 4243 payload 824 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 1 -- 172.21.15.110:0/2622855084 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0b4c80 msgr2=0x7f3bec13a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0b4c80 0x7f3bec13a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 1 -- 172.21.15.110:0/2622855084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bec142060 msgr2=0x7f3bec1365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bec142060 0x7f3bec1365d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 10 monclient: _finish_hunting 0 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 1 monclient: found mon.noname-b 2024-01-27T03:43:11.281 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:11.282 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:11.282 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.273+0000 7f3bf0801640 1 -- 172.21.15.110:0/2622855084 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc0094d0 con 0x7f3bec137180 2024-01-27T03:43:11.286 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.275+0000 7f3bf0801640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec137180 0x7f3bec1375d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3be400c6e0 tx=0x7f3be400cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:11.286 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.275+0000 7f3be8ff9640 10 client.?.objecter ms_handle_connect 0x7f3bec137180 2024-01-27T03:43:11.286 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.275+0000 7f3be8ff9640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:11.287 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 1 -- 172.21.15.110:0/2622855084 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4003b70 con 0x7f3bec137180 2024-01-27T03:43:11.287 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:11.287 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:11.287 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 10 monclient: dump: 2024-01-27T03:43:11.287 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:11.287 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:11.287 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:11.287 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:11.287 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:11.288 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:11.288 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:11.288 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:11.288 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:11.288 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:11.288 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 10 monclient: _finish_auth 0 2024-01-27T03:43:11.289 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 10 monclient: _check_auth_tickets 2024-01-27T03:43:11.289 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:11.289 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 1 -- 172.21.15.110:0/2622855084 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be4003cf0 con 0x7f3bec137180 2024-01-27T03:43:11.289 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bf2264640 5 monclient: authenticate success, global_id 4243 2024-01-27T03:43:11.289 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bf2264640 10 monclient: _renew_subs 2024-01-27T03:43:11.289 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bf2264640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:11.290 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec136c60 con 0x7f3bec137180 2024-01-27T03:43:11.290 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:11.290 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 1 -- 172.21.15.110:0/2622855084 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4005310 con 0x7f3bec137180 2024-01-27T03:43:11.290 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:11.290 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:11.290 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3be8ff9640 10 monclient: dump: 2024-01-27T03:43:11.291 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:11.291 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:11.291 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:11.291 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:11.291 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:11.291 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:11.292 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:11.292 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:11.292 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:11.292 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:11.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bea7fc640 4 set_mon_vals no callback set 2024-01-27T03:43:11.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bf2264640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:11.292 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bf2264640 10 monclient: _renew_subs 2024-01-27T03:43:11.293 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bf2264640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:11.293 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bec1dac60 con 0x7f3bec137180 2024-01-27T03:43:11.293 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.278+0000 7f3bf2264640 1 librados: init done 2024-01-27T03:43:11.293 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.279+0000 7f3bf2264640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:11.293 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.279+0000 7f3bf2264640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:11.293 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.279+0000 7f3bf2264640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:11.293 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.279+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec0b3920 con 0x7f3bec137180 2024-01-27T03:43:11.300 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.298+0000 7f3be8ff9640 1 -- 172.21.15.110:0/2622855084 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 4) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f3be4005470 con 0x7f3bec137180 2024-01-27T03:43:11.301 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.298+0000 7f3be8ff9640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f3bd806c910 0x7f3bd806edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:11.301 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.298+0000 7f3bebfff640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f3bd806c910 0x7f3bd806edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:11.301 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.299+0000 7f3bebfff640 10 monclient: get_auth_request con 0x7f3bd806c910 auth_method 0 2024-01-27T03:43:11.301 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.299+0000 7f3be8ff9640 10 client.4243.objecter ms_handle_connect 0x7f3bd806c910 2024-01-27T03:43:11.301 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.299+0000 7f3bebfff640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f3bd806c910 0x7f3bd806edd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc036000 tx=0x7f3bdc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:11.309 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.300+0000 7f3be8ff9640 1 -- 172.21.15.110:0/2622855084 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2511+0+0 (secure 0 0 0) 0x7f3be4058af0 con 0x7f3bec137180 2024-01-27T03:43:11.309 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.300+0000 7f3be8ff9640 10 client.4243.objecter ms_dispatch 0x7f3bec000b90 osd_map(14..14 src has 1..14) v4 2024-01-27T03:43:11.309 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.300+0000 7f3be8ff9640 3 client.4243.objecter handle_osd_map got epochs [14,14] > 0 2024-01-27T03:43:11.309 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.300+0000 7f3be8ff9640 3 client.4243.objecter handle_osd_map decoding full epoch 14 2024-01-27T03:43:11.309 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.300+0000 7f3be8ff9640 20 client.4243.objecter dump_active .. 0 homeless 2024-01-27T03:43:11.501 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:43:11.483+0000 10253640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-27T03:43:11.551 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.549+0000 7f3be8ff9640 1 -- 172.21.15.110:0/2622855084 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f3be4054d90 con 0x7f3bec137180 2024-01-27T03:43:11.551 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.549+0000 7f3be8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:11.551 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.549+0000 7f3be8ff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:11.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-27T03:43:11.547+0000 10253640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-27T03:43:11.828 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.827+0000 7f3bf2264640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:11.829 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.827+0000 7f3bf2264640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:11.829 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.827+0000 7f3bf2264640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:11.829 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:11.827+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bec04f2a0 con 0x7f3bec137180 2024-01-27T03:43:12.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.015+0000 7f3be8ff9640 1 -- 172.21.15.110:0/2622855084 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+8040 (secure 0 0 0) 0x7f3be4003e60 con 0x7f3bec137180 2024-01-27T03:43:12.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.015+0000 7f3be8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:12.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.015+0000 7f3be8ff9640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:12.017 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-27T03:43:12.017 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":15,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:43:11.657307+0000","last_up_change":"2024-01-27T03:43:11.657307+0000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6803","nonce":946981202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6805","nonce":946981202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6809","nonce":946981202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6807","nonce":946981202}]},"public_addr":"172.21.15.162:6803/946981202","cluster_addr":"172.21.15.162:6805/946981202","heartbeat_back_addr":"172.21.15.162:6809/946981202","heartbeat_front_addr":"172.21.15.162:6807/946981202","state":["exists","up"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6819","nonce":68785612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6821","nonce":68785612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6825","nonce":68785612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6823","nonce":68785612}]},"public_addr":"172.21.15.162:6819/68785612","cluster_addr":"172.21.15.162:6821/68785612","heartbeat_back_addr":"172.21.15.162:6825/68785612","heartbeat_front_addr":"172.21.15.162:6823/68785612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:07.959074+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:43:12.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 10 librados: watch_flush enter 2024-01-27T03:43:12.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 10 librados: watch_flush exit 2024-01-27T03:43:12.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 20 client.4243.objecter shutdown clearing up homeless session... 2024-01-27T03:43:12.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 10 client.4243.objecter successfully canceled tick 2024-01-27T03:43:12.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f3bd806c910 msgr2=0x7f3bd806edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:12.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f3bd806c910 0x7f3bd806edd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc036000 tx=0x7f3bdc008030 comp rx=0 tx=0).stop 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 10 monclient: shutdown 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec137180 msgr2=0x7f3bec1375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec137180 0x7f3bec1375d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3be400c6e0 tx=0x7f3be400cba0 comp rx=0 tx=0).stop 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 shutdown_connections 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f3bd806c910 0x7f3bd806edd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bec137180 0x7f3bec1375d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3bec142060 0x7f3bec1365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 --2- 172.21.15.110:0/2622855084 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3bec0b4c80 0x7f3bec13a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 >> 172.21.15.110:0/2622855084 conn(0x7f3bec071f90 msgr2=0x7f3bec142470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 shutdown_connections 2024-01-27T03:43:12.021 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 -- 172.21.15.110:0/2622855084 wait complete. 2024-01-27T03:43:12.022 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:12.018+0000 7f3bf2264640 1 librados: shutdown 2024-01-27T03:43:12.029 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-01-27T03:43:16.878 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:16.826+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:16.889 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:16.878+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:16.914 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:16.897+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:16.925 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:16.914+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.092 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:16.894+0000 5dc2480 -1 bdev(0x1648fb30 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-01-27T03:43:17.092 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:16.966+0000 5dc2480 -1 bdev(0x1648fb30 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-27T03:43:17.092 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:16.980+0000 5dc2480 -1 bdev(0x1648fb30 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-27T03:43:17.092 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:16.992+0000 5dc2480 -1 bdev(0x1648fb30 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-27T03:43:17.098 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:17.035+0000 5dc2480 -1 bdev(0x1648fb30 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-27T03:43:17.115 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:16.927+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-01-27T03:43:17.116 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.072+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.302 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.118+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-01-27T03:43:17.321 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.151+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.321 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.163+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.321 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.247+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.321 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.260+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.496 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.302+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.514 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:17.098+0000 5dc2480 -1 bdev(0x1648fb30 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-01-27T03:43:17.540 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.313+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.540 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.438+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.541 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.496+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.542 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.542+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:17.648 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:17.515+0000 5dc2480 -1 bdev(0x1648fb30 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-27T03:43:17.741 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:17.649+0000 5dc2480 -1 bdev(0x1648fb30 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-01-27T03:43:17.788 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:17.743+0000 5dc2480 -1 bdev(0x1648fb30 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-27T03:43:17.876 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:17.764+0000 5dc2480 -1 bdev(0x164ae450 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-27T03:43:18.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.100+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.167+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.178+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.233+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.310 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.246+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.294+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.800 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:18.795+0000 c494640 -1 osd.4 0 waiting for initial osdmap 2024-01-27T03:43:18.805 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.310+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.821 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.733+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.836 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-27T03:43:18.859 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.805+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.906 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:18.901+0000 c494640 -1 osd.6 0 waiting for initial osdmap 2024-01-27T03:43:18.913 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:18.821+0000 5dc2480 -1 bdev(0x18ee5660 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-01-27T03:43:18.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.938+0000 7f163f821640 10 monclient: get_monmap_and_config 2024-01-27T03:43:18.939 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.938+0000 7f163f821640 10 monclient: build_initial_monmap 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.938+0000 7f163f821640 10 monclient: monmap: 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:43:18.939974+0000 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:43:18.939974+0000 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:18.940 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:18.941 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.939+0000 7f163f821640 1 Processor -- start 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.939+0000 7f163f821640 1 -- start start 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.939+0000 7f163f821640 10 monclient: init 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16380b4c80 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f16380a9e20 0x7f1638142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 10 monclient(hunting): picked mon.noname-c con 0x7f16380a9e20 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f163813fa10 0x7f16381425a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 10 monclient(hunting): picked mon.noname-a con 0x7f163813fa10 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16380a99d0 con 0x7f16380b4c80 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16380b5060 con 0x7f163813fa10 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:18.942 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16380b3ed0 con 0x7f16380a9e20 2024-01-27T03:43:18.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:18.943 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.940+0000 7f163f821640 10 monclient(hunting): authenticate will time out at 1482.294346s 2024-01-27T03:43:18.989 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.987+0000 7f163cdbc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f16380a9e20 0x7f1638142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:18.989 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.987+0000 7f163d5bd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f16380b4c80 0x7f163813f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:41736/0 (socket says 172.21.15.110:41736) 2024-01-27T03:43:18.989 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.987+0000 7f163d5bd640 1 -- 172.21.15.110:0/1973731249 learned_addr learned my addr 172.21.15.110:0/1973731249 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:18.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.987+0000 7f163cdbc640 10 monclient(hunting): get_auth_request con 0x7f16380a9e20 auth_method 0 2024-01-27T03:43:18.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.987+0000 7f163cdbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:18.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.987+0000 7f163cdbc640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:18.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.987+0000 7f163cdbc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:18.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.988+0000 7f163cdbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:18.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.988+0000 7f163cdbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:18.990 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.988+0000 7f163cdbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:18.991 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.990+0000 7f163cdbc640 10 monclient(hunting): handle_auth_done global_id 4252 payload 274 2024-01-27T03:43:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.990+0000 7f163cdbc640 1 -- 172.21.15.110:0/1973731249 >> v2:172.21.15.110:3300/0 conn(0x7f163813fa10 msgr2=0x7f16381425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.990+0000 7f163cdbc640 1 --2- 172.21.15.110:0/1973731249 >> v2:172.21.15.110:3300/0 conn(0x7f163813fa10 0x7f16381425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.990+0000 7f163cdbc640 1 -- 172.21.15.110:0/1973731249 >> v1:172.21.15.162:6789/0 conn(0x7f16380b4c80 legacy=0x7f163813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.990+0000 7f163cdbc640 10 monclient: _finish_hunting 0 2024-01-27T03:43:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.990+0000 7f163cdbc640 1 monclient: found mon.noname-c 2024-01-27T03:43:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.990+0000 7f163cdbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.990+0000 7f163cdbc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:18.992 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.990+0000 7f163cdbc640 1 -- 172.21.15.110:0/1973731249 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16380b3b60 con 0x7f16380a9e20 2024-01-27T03:43:18.995 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.994+0000 7f163cdbc640 1 --2- 172.21.15.110:0/1973731249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16380a9e20 0x7f1638142060 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16380ab2f0 tx=0x7f162802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b1e0f9698acdf3f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.997+0000 7f162ffff640 1 -- 172.21.15.110:0/1973731249 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628031830 con 0x7f16380a9e20 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.997+0000 7f162ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.997+0000 7f162ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.997+0000 7f162ffff640 10 monclient: dump: 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:18.999 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.997+0000 7f162ffff640 10 monclient: _finish_auth 0 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.997+0000 7f162ffff640 10 monclient: _check_auth_tickets 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.997+0000 7f162ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.997+0000 7f162ffff640 1 -- 172.21.15.110:0/1973731249 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1628031990 con 0x7f16380a9e20 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.997+0000 7f162ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f162ffff640 1 -- 172.21.15.110:0/1973731249 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628031d20 con 0x7f16380a9e20 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f162ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f162ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f162ffff640 10 monclient: dump: 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:19.000 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:19.001 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:19.001 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:19.001 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:19.001 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:19.001 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:19.001 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:19.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 5 monclient: authenticate success, global_id 4252 2024-01-27T03:43:19.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:19.001 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 4 set_mon_vals no callback set 2024-01-27T03:43:19.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 10 monclient: shutdown 2024-01-27T03:43:19.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:19.004 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 1 -- 172.21.15.110:0/1973731249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16380a9e20 msgr2=0x7f1638142060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 1 --2- 172.21.15.110:0/1973731249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16380a9e20 0x7f1638142060 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16380ab2f0 tx=0x7f162802b8d0 comp rx=0 tx=0).stop 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 1 -- 172.21.15.110:0/1973731249 shutdown_connections 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 1 --2- 172.21.15.110:0/1973731249 >> v2:172.21.15.110:3300/0 conn(0x7f163813fa10 0x7f16381425a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 1 --2- 172.21.15.110:0/1973731249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16380a9e20 0x7f1638142060 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 1 -- 172.21.15.110:0/1973731249 >> 172.21.15.110:0/1973731249 conn(0x7f1638071f90 msgr2=0x7f16380723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.998+0000 7f163f821640 1 -- 172.21.15.110:0/1973731249 shutdown_connections 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.999+0000 7f163f821640 1 -- 172.21.15.110:0/1973731249 wait complete. 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.999+0000 7f163f821640 10 monclient: build_initial_monmap 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:18.999+0000 7f163f821640 10 monclient: monmap: 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:43:19.005 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.000+0000 7f163f821640 1 librados: starting msgr at 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.000+0000 7f163f821640 1 librados: starting objecter 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.000+0000 7f163f821640 1 Processor -- start 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.000+0000 7f163f821640 1 -- start start 2024-01-27T03:43:19.006 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.000+0000 7f163f821640 1 librados: setting wanted keys 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.000+0000 7f163f821640 1 librados: calling monclient init 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.000+0000 7f163f821640 10 monclient: init 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16380b4c80 0x7f16380ac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient(hunting): picked mon.noname-c con 0x7f16380b4c80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f163813fa10 0x7f16381db7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient(hunting): picked mon.noname-a con 0x7f163813fa10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16381dbcf0 0x7f16381de9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient(hunting): picked mon.noname-b con 0x7f16381dbcf0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16380acff0 con 0x7f163813fa10 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:19.007 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16380accf0 con 0x7f16381dbcf0 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16380abed0 con 0x7f16380b4c80 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.001+0000 7f163f821640 10 monclient(hunting): authenticate will time out at 1482.355728s 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.002+0000 7f163ddbe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16381dbcf0 0x7f16381de9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.002+0000 7f163ddbe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16381dbcf0 0x7f16381de9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:57474/0 (socket says 172.21.15.110:57474) 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.002+0000 7f163ddbe640 1 -- 172.21.15.110:0/2132387961 learned_addr learned my addr 172.21.15.110:0/2132387961 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.002+0000 7f163ddbe640 10 monclient(hunting): get_auth_request con 0x7f16381dbcf0 auth_method 0 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.002+0000 7f163ddbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.002+0000 7f163ddbe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.002+0000 7f163ddbe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.002+0000 7f163ddbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.003+0000 7f163ddbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:19.008 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.003+0000 7f163ddbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 10 monclient(hunting): handle_auth_done global_id 4255 payload 824 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 1 -- 172.21.15.110:0/2132387961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16380b4c80 msgr2=0x7f16380ac7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16380b4c80 0x7f16380ac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 1 -- 172.21.15.110:0/2132387961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f163813fa10 msgr2=0x7f16381db7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f163813fa10 0x7f16381db7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 10 monclient: _finish_hunting 0 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 1 monclient: found mon.noname-b 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.005+0000 7f163ddbe640 1 -- 172.21.15.110:0/2132387961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1628009400 con 0x7f16381dbcf0 2024-01-27T03:43:19.010 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.008+0000 7f163ddbe640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16381dbcf0 0x7f16381de9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f163000b970 tx=0x7f163000be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:19.011 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.008+0000 7f162dffb640 10 client.?.objecter ms_handle_connect 0x7f16381dbcf0 2024-01-27T03:43:19.011 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.008+0000 7f162dffb640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:19.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162dffb640 1 -- 172.21.15.110:0/2132387961 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163000cb40 con 0x7f16381dbcf0 2024-01-27T03:43:19.015 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162dffb640 10 monclient: dump: 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162dffb640 10 monclient: _finish_auth 0 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162dffb640 10 monclient: _check_auth_tickets 2024-01-27T03:43:19.016 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:19.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162dffb640 1 -- 172.21.15.110:0/2132387961 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163000ccc0 con 0x7f16381dbcf0 2024-01-27T03:43:19.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:19.017 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.014+0000 7f162f7fe640 4 set_mon_vals no callback set 2024-01-27T03:43:19.018 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.015+0000 7f163f821640 5 monclient: authenticate success, global_id 4255 2024-01-27T03:43:19.018 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.015+0000 7f162dffb640 1 -- 172.21.15.110:0/2132387961 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1630014680 con 0x7f16381dbcf0 2024-01-27T03:43:19.018 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.015+0000 7f163f821640 10 monclient: _renew_subs 2024-01-27T03:43:19.018 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.015+0000 7f163f821640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:19.018 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.015+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16380ab610 con 0x7f16381dbcf0 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.015+0000 7f163f821640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.015+0000 7f162dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.015+0000 7f162dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.015+0000 7f162dffb640 10 monclient: dump: 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.016+0000 7f163f821640 10 monclient: _renew_subs 2024-01-27T03:43:19.019 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.016+0000 7f163f821640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:19.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.016+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16381dc180 con 0x7f16381dbcf0 2024-01-27T03:43:19.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.016+0000 7f163f821640 1 librados: init done 2024-01-27T03:43:19.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.016+0000 7f163f821640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:19.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.016+0000 7f163f821640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:19.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.016+0000 7f163f821640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:19.020 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.016+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16381dc3d0 con 0x7f16381dbcf0 2024-01-27T03:43:19.046 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.045+0000 7f162dffb640 1 -- 172.21.15.110:0/2132387961 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 4) v1 ==== 89194+0+0 (secure 0 0 0) 0x7f163001d3b0 con 0x7f16381dbcf0 2024-01-27T03:43:19.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.046+0000 7f162dffb640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f161c06c960 0x7f161c06ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:19.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.046+0000 7f163d5bd640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f161c06c960 0x7f161c06ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:19.047 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.046+0000 7f163d5bd640 10 monclient: get_auth_request con 0x7f161c06c960 auth_method 0 2024-01-27T03:43:19.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.047+0000 7f163d5bd640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f161c06c960 0x7f161c06ee20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f16340015f0 tx=0x7f1634007a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:19.048 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.047+0000 7f162dffb640 10 client.4255.objecter ms_handle_connect 0x7f161c06c960 2024-01-27T03:43:19.305 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.304+0000 7f162dffb640 1 -- 172.21.15.110:0/2132387961 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3737+0+0 (secure 0 0 0) 0x7f163000ce30 con 0x7f16381dbcf0 2024-01-27T03:43:19.306 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.304+0000 7f162dffb640 10 client.4255.objecter ms_dispatch 0x7f1638000b90 osd_map(19..19 src has 1..19) v4 2024-01-27T03:43:19.306 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.304+0000 7f162dffb640 3 client.4255.objecter handle_osd_map got epochs [19,19] > 0 2024-01-27T03:43:19.306 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.304+0000 7f162dffb640 3 client.4255.objecter handle_osd_map decoding full epoch 19 2024-01-27T03:43:19.306 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.304+0000 7f162dffb640 20 client.4255.objecter dump_active .. 0 homeless 2024-01-27T03:43:19.322 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:43:19.309+0000 1425b640 -1 osd.4 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-27T03:43:19.325 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.324+0000 7f162dffb640 1 -- 172.21.15.110:0/2132387961 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f1630058500 con 0x7f16381dbcf0 2024-01-27T03:43:19.325 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.324+0000 7f162dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:19.325 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.324+0000 7f162dffb640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:19.502 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:43:19.488+0000 1425b640 -1 osd.6 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-27T03:43:19.617 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.616+0000 7f163f821640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:19.617 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.616+0000 7f163f821640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:19.617 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.616+0000 7f163f821640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:19.618 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.616+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f163804f2a0 con 0x7f16381dbcf0 2024-01-27T03:43:19.718 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.718+0000 7f162dffb640 1 -- 172.21.15.110:0/2132387961 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f1630054da0 con 0x7f16381dbcf0 2024-01-27T03:43:19.954 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.952+0000 7f162dffb640 1 -- 172.21.15.110:0/2132387961 <== mon.1 v2:172.21.15.162:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+10886 (secure 0 0 0) 0x7f163003e640 con 0x7f16381dbcf0 2024-01-27T03:43:19.954 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.953+0000 7f162dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:19.954 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.953+0000 7f162dffb640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:19.954 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-27T03:43:19.954 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":19,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:43:16.323587+0000","last_up_change":"2024-01-27T03:43:12.684115+0000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-27T03:43:13.074487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6817","nonce":1545415171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6819","nonce":1545415171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6824","nonce":1545415171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6821","nonce":1545415171}]},"public_addr":"172.21.15.110:6817/1545415171","cluster_addr":"172.21.15.110:6819/1545415171","heartbeat_back_addr":"172.21.15.110:6824/1545415171","heartbeat_front_addr":"172.21.15.110:6821/1545415171","state":["exists","up"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6801","nonce":2374111370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6803","nonce":2374111370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6807","nonce":2374111370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6805","nonce":2374111370}]},"public_addr":"172.21.15.110:6801/2374111370","cluster_addr":"172.21.15.110:6803/2374111370","heartbeat_back_addr":"172.21.15.110:6807/2374111370","heartbeat_front_addr":"172.21.15.110:6805/2374111370","state":["exists","up"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6803","nonce":946981202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6805","nonce":946981202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6809","nonce":946981202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6807","nonce":946981202}]},"public_addr":"172.21.15.162:6803/946981202","cluster_addr":"172.21.15.162:6805/946981202","heartbeat_back_addr":"172.21.15.162:6809/946981202","heartbeat_front_addr":"172.21.15.162:6807/946981202","state":["exists","up"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6819","nonce":68785612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6821","nonce":68785612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6825","nonce":68785612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6823","nonce":68785612}]},"public_addr":"172.21.15.162:6819/68785612","cluster_addr":"172.21.15.162:6821/68785612","heartbeat_back_addr":"172.21.15.162:6825/68785612","heartbeat_front_addr":"172.21.15.162:6823/68785612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.313498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.904665+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.104354+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:07.959074+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:43:19.957 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.955+0000 7f163f821640 10 librados: watch_flush enter 2024-01-27T03:43:19.957 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.955+0000 7f163f821640 10 librados: watch_flush exit 2024-01-27T03:43:19.957 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.955+0000 7f163f821640 20 client.4255.objecter shutdown clearing up homeless session... 2024-01-27T03:43:19.957 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.955+0000 7f163f821640 10 client.4255.objecter successfully canceled tick 2024-01-27T03:43:19.957 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f161c06c960 msgr2=0x7f161c06ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f161c06c960 0x7f161c06ee20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f16340015f0 tx=0x7f1634007a40 comp rx=0 tx=0).stop 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 10 monclient: shutdown 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16381dbcf0 msgr2=0x7f16381de9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16381dbcf0 0x7f16381de9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f163000b970 tx=0x7f163000be30 comp rx=0 tx=0).stop 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 shutdown_connections 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f161c06c960 0x7f161c06ee20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16381dbcf0 0x7f16381de9b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f163813fa10 0x7f16381db7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 --2- 172.21.15.110:0/2132387961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16380b4c80 0x7f16380ac7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 >> 172.21.15.110:0/2132387961 conn(0x7f1638071f90 msgr2=0x7f16380b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 shutdown_connections 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 -- 172.21.15.110:0/2132387961 wait complete. 2024-01-27T03:43:19.958 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:19.956+0000 7f163f821640 1 librados: shutdown 2024-01-27T03:43:19.967 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-01-27T03:43:20.140 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:43:20.134+0000 c494640 -1 osd.2 0 waiting for initial osdmap 2024-01-27T03:43:20.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:20.241+0000 c494640 -1 osd.1 0 waiting for initial osdmap 2024-01-27T03:43:20.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:43:20.622+0000 1425b640 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-27T03:43:20.782 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:43:20.766+0000 1425b640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-27T03:43:23.268 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:09.224 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:23.334 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:09.290 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:23.337 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:01:09.282 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:23.376 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:09.332 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:23.397 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:01:09.343 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:23.447 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:09.403 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:23.613 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:09.569 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:23.650 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:01:09.595 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:23.701 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:09.657 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:26.774 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-27T03:43:26.873 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.872+0000 7fb77c890640 10 monclient: get_monmap_and_config 2024-01-27T03:43:26.873 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.872+0000 7fb77c890640 10 monclient: build_initial_monmap 2024-01-27T03:43:26.873 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.872+0000 7fb77c890640 10 monclient: monmap: 2024-01-27T03:43:26.873 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:43:26.873 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:43:26.873262+0000 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:43:26.873262+0000 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:26.874 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:26.875 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.872+0000 7fb77c890640 1 Processor -- start 2024-01-27T03:43:26.875 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.872+0000 7fb77c890640 1 -- start start 2024-01-27T03:43:26.875 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.872+0000 7fb77c890640 10 monclient: init 2024-01-27T03:43:26.875 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:26.875 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-01-27T03:43:26.875 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7780b4c80 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:26.875 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb7780a9e20 0x7fb778142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:26.875 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 10 monclient(hunting): picked mon.noname-a con 0x7fb7780a9e20 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:26.876 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb77813fa10 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:26.876 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7780a99d0 con 0x7fb77813fa10 2024-01-27T03:43:26.876 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7780b3ed0 con 0x7fb7780b4c80 2024-01-27T03:43:26.876 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:26.876 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7780b5060 con 0x7fb7780a9e20 2024-01-27T03:43:26.876 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:26.876 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.873+0000 7fb77c890640 10 monclient(hunting): authenticate will time out at 1490.227506s 2024-01-27T03:43:26.877 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.874+0000 7fb77659c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb7780b4c80 0x7fb77813f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:36706/0 (socket says 172.21.15.110:36706) 2024-01-27T03:43:26.877 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.874+0000 7fb77659c640 1 -- 172.21.15.110:0/1560913619 learned_addr learned my addr 172.21.15.110:0/1560913619 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:26.877 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.874+0000 7fb775d9b640 1 --2- 172.21.15.110:0/1560913619 >> v2:172.21.15.110:3300/0 conn(0x7fb7780a9e20 0x7fb778142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:26.877 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.875+0000 7fb775d9b640 10 monclient(hunting): get_auth_request con 0x7fb7780a9e20 auth_method 0 2024-01-27T03:43:26.877 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.875+0000 7fb775d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:26.877 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.875+0000 7fb775d9b640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:26.877 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.875+0000 7fb775d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:26.878 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.876+0000 7fb775d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:26.878 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.876+0000 7fb775d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:26.878 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.876+0000 7fb775d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:26.878 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.877+0000 7fb77559a640 1 -- 172.21.15.110:0/1560913619 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755320890 0 0) 0x7fb7780b3ed0 con 0x7fb7780b4c80 2024-01-27T03:43:26.878 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.877+0000 7fb77559a640 10 monclient(hunting): _negotiate 2024-01-27T03:43:26.878 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.877+0000 7fb77559a640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:26.878 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.877+0000 7fb77559a640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:26.878 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.877+0000 7fb77559a640 10 monclient(hunting): my global_id is 4261 2024-01-27T03:43:26.878 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.877+0000 7fb77559a640 1 -- 172.21.15.110:0/1560913619 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb760003520 con 0x7fb7780b4c80 2024-01-27T03:43:26.879 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.878+0000 7fb775d9b640 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-01-27T03:43:26.880 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.878+0000 7fb775d9b640 1 -- 172.21.15.110:0/1560913619 >> v1:172.21.15.162:6789/0 conn(0x7fb7780b4c80 legacy=0x7fb77813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:26.880 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.878+0000 7fb775d9b640 1 -- 172.21.15.110:0/1560913619 >> v1:172.21.15.110:6789/0 conn(0x7fb77813fa10 legacy=0x7fb77814c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:26.880 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.878+0000 7fb775d9b640 10 monclient: _finish_hunting 0 2024-01-27T03:43:26.880 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.878+0000 7fb775d9b640 1 monclient: found mon.noname-a 2024-01-27T03:43:26.880 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.878+0000 7fb775d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:26.880 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.878+0000 7fb775d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:26.880 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.878+0000 7fb775d9b640 1 -- 172.21.15.110:0/1560913619 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780b3b60 con 0x7fb7780a9e20 2024-01-27T03:43:26.883 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.882+0000 7fb775d9b640 1 --2- 172.21.15.110:0/1560913619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7780a9e20 0x7fb778142060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0097a0 tx=0x7fb76c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69cf32b21cd7723b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:26.887 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 1 -- 172.21.15.110:0/1560913619 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c030610 con 0x7fb7780a9e20 2024-01-27T03:43:26.887 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:26.888 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:26.888 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 10 monclient: dump: 2024-01-27T03:43:26.888 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:26.888 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:26.888 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:26.888 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:26.888 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:26.889 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:26.889 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:26.889 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:26.889 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:26.889 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:26.889 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 10 monclient: _finish_auth 0 2024-01-27T03:43:26.889 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 10 monclient: _check_auth_tickets 2024-01-27T03:43:26.889 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:26.890 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 1 -- 172.21.15.110:0/1560913619 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c030770 con 0x7fb7780a9e20 2024-01-27T03:43:26.890 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 5 monclient: authenticate success, global_id 4266 2024-01-27T03:43:26.890 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:26.890 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:26.890 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:26.890 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 1 -- 172.21.15.110:0/1560913619 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c030b00 con 0x7fb7780a9e20 2024-01-27T03:43:26.890 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 4 set_mon_vals no callback set 2024-01-27T03:43:26.890 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:26.890 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:26.891 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77559a640 10 monclient: dump: 2024-01-27T03:43:26.891 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:26.891 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:26.891 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:26.891 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:26.891 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:26.891 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:26.891 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:26.891 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:26.892 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:26.892 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:26.897 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 10 monclient: shutdown 2024-01-27T03:43:26.897 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 1 -- 172.21.15.110:0/1560913619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7780a9e20 msgr2=0x7fb778142060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 1 --2- 172.21.15.110:0/1560913619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7780a9e20 0x7fb778142060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0097a0 tx=0x7fb76c02b960 comp rx=0 tx=0).stop 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 1 -- 172.21.15.110:0/1560913619 shutdown_connections 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 1 --2- 172.21.15.110:0/1560913619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7780a9e20 0x7fb778142060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.886+0000 7fb77c890640 1 -- 172.21.15.110:0/1560913619 >> 172.21.15.110:0/1560913619 conn(0x7fb778071f90 msgr2=0x7fb7780723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.887+0000 7fb77c890640 1 -- 172.21.15.110:0/1560913619 shutdown_connections 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.887+0000 7fb77c890640 1 -- 172.21.15.110:0/1560913619 wait complete. 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.887+0000 7fb77c890640 10 monclient: build_initial_monmap 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.887+0000 7fb77c890640 10 monclient: monmap: 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-01-27T03:43:26.898 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 librados: starting msgr at 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 librados: starting objecter 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 Processor -- start 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 -- start start 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 librados: setting wanted keys 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 librados: calling monclient init 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient: init 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:43:26.899 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7780b4c80 0x7fb7781407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient(hunting): picked mon.noname-b con 0x7fb7780b4c80 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77813fa10 0x7fb7781d8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient(hunting): picked mon.noname-c con 0x7fb77813fa10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb778144ba0 0x7fb778145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient(hunting): picked mon.noname-a con 0x7fb778144ba0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb778145560 con 0x7fb778144ba0 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb778146260 con 0x7fb7780b4c80 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:26.900 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7781d7d60 con 0x7fb77813fa10 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.888+0000 7fb77c890640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb77c890640 10 monclient(hunting): authenticate will time out at 1490.242963s 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb776d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb778144ba0 0x7fb778145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb775d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77813fa10 0x7fb7781d8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb77659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7780b4c80 0x7fb7781407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb776d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb778144ba0 0x7fb778145020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33768/0 (socket says 172.21.15.110:33768) 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb776d9d640 1 -- 172.21.15.110:0/1008197205 learned_addr learned my addr 172.21.15.110:0/1008197205 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb776d9d640 10 monclient(hunting): get_auth_request con 0x7fb778144ba0 auth_method 0 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb776d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb776d9d640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb776d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb77659c640 10 monclient(hunting): get_auth_request con 0x7fb7780b4c80 auth_method 0 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb77659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb77659c640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.889+0000 7fb77659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:26.901 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb775d9b640 10 monclient(hunting): get_auth_request con 0x7fb77813fa10 auth_method 0 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb775d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb775d9b640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb775d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb776d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb776d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb776d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb77659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb77659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb77659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb775d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb775d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.890+0000 7fb775d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2024-01-27T03:43:26.902 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 1 -- 172.21.15.110:0/1008197205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77813fa10 msgr2=0x7fb7781d8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77813fa10 0x7fb7781d8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 1 -- 172.21.15.110:0/1008197205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7780b4c80 msgr2=0x7fb7781407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7780b4c80 0x7fb7781407b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 10 monclient: _finish_hunting 0 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 1 monclient: found mon.noname-a 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.893+0000 7fb776d9d640 1 -- 172.21.15.110:0/1008197205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c009400 con 0x7fb778144ba0 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.895+0000 7fb776d9d640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb778144ba0 0x7fb778145020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb77000c3c0 tx=0x7fb77000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.895+0000 7fb766ffd640 10 client.?.objecter ms_handle_connect 0x7fb778144ba0 2024-01-27T03:43:26.903 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.895+0000 7fb766ffd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb766ffd640 1 -- 172.21.15.110:0/1008197205 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700052d0 con 0x7fb778144ba0 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb766ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb766ffd640 10 monclient: dump: 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:26.905 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb766ffd640 10 monclient: _finish_auth 0 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb766ffd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb77c890640 5 monclient: authenticate success, global_id 4272 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb77c890640 10 monclient: _renew_subs 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb77c890640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7781d87c0 con 0x7fb778144ba0 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb77c890640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb77c890640 10 monclient: _renew_subs 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb77c890640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb778135eb0 con 0x7fb778144ba0 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.897+0000 7fb77c890640 1 librados: init done 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.898+0000 7fb77c890640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:26.906 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.898+0000 7fb77c890640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.898+0000 7fb77c890640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.898+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7781d7b80 con 0x7fb778144ba0 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.899+0000 7fb766ffd640 1 -- 172.21.15.110:0/1008197205 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb770005430 con 0x7fb778144ba0 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.899+0000 7fb766ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.899+0000 7fb766ffd640 1 -- 172.21.15.110:0/1008197205 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770007220 con 0x7fb778144ba0 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.899+0000 7fb766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.899+0000 7fb766ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.899+0000 7fb766ffd640 10 monclient: dump: 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:26.907 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:26.908 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:26.908 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:26.908 INFO:teuthology.misc.health.smithi110.stderr: 2024-01-27T03:43:26.908 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:26.899+0000 7fb774d99640 4 set_mon_vals no callback set 2024-01-27T03:43:27.142 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.141+0000 7fb766ffd640 1 -- 172.21.15.110:0/1008197205 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fb770007380 con 0x7fb778144ba0 2024-01-27T03:43:27.143 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.142+0000 7fb766ffd640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:27.143 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.142+0000 7fb77659c640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:27.143 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.142+0000 7fb77659c640 10 monclient: get_auth_request con 0x7fb75806c9a0 auth_method 0 2024-01-27T03:43:27.144 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.143+0000 7fb77659c640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb768001560 tx=0x7fb768001a90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:27.144 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.143+0000 7fb766ffd640 10 client.4272.objecter ms_handle_connect 0x7fb75806c9a0 2024-01-27T03:43:27.158 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.154+0000 7fb766ffd640 1 -- 172.21.15.110:0/1008197205 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb770057cf0 con 0x7fb778144ba0 2024-01-27T03:43:27.158 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.154+0000 7fb766ffd640 10 client.4272.objecter ms_dispatch 0x7fb778000b90 osd_map(23..23 src has 1..23) v4 2024-01-27T03:43:27.158 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.154+0000 7fb766ffd640 3 client.4272.objecter handle_osd_map got epochs [23,23] > 0 2024-01-27T03:43:27.158 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.154+0000 7fb766ffd640 3 client.4272.objecter handle_osd_map decoding full epoch 23 2024-01-27T03:43:27.158 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.154+0000 7fb766ffd640 20 client.4272.objecter dump_active .. 0 homeless 2024-01-27T03:43:27.159 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.154+0000 7fb766ffd640 1 -- 172.21.15.110:0/1008197205 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fb770053f90 con 0x7fb778144ba0 2024-01-27T03:43:27.159 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.154+0000 7fb766ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:27.159 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.154+0000 7fb766ffd640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:27.334 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.333+0000 7fb77c890640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:27.334 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.333+0000 7fb77c890640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:27.334 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.333+0000 7fb77c890640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:27.334 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.333+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb77804f2a0 con 0x7fb778144ba0 2024-01-27T03:43:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.421+0000 7fb766ffd640 1 -- 172.21.15.110:0/1008197205 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+12864 (secure 0 0 0) 0x7fb770017090 con 0x7fb778144ba0 2024-01-27T03:43:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.421+0000 7fb766ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:27.422 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.421+0000 7fb766ffd640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:27.423 INFO:teuthology.misc.health.smithi110.stdout: 2024-01-27T03:43:27.423 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":23,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:43:23.834696+0000","last_up_change":"2024-01-27T03:43:21.487334+0000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-27T03:43:13.074487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6817","nonce":1545415171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6819","nonce":1545415171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6824","nonce":1545415171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6821","nonce":1545415171}]},"public_addr":"172.21.15.110:6817/1545415171","cluster_addr":"172.21.15.110:6819/1545415171","heartbeat_back_addr":"172.21.15.110:6824/1545415171","heartbeat_front_addr":"172.21.15.110:6821/1545415171","state":["exists","up"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6823","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6825","nonce":518896551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6827","nonce":518896551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6831","nonce":518896551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6829","nonce":518896551}]},"public_addr":"172.21.15.110:6825/518896551","cluster_addr":"172.21.15.110:6827/518896551","heartbeat_back_addr":"172.21.15.110:6831/518896551","heartbeat_front_addr":"172.21.15.110:6829/518896551","state":["exists","up"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6809","nonce":3683566991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6811","nonce":3683566991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6815","nonce":3683566991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6813","nonce":3683566991}]},"public_addr":"172.21.15.110:6809/3683566991","cluster_addr":"172.21.15.110:6811/3683566991","heartbeat_back_addr":"172.21.15.110:6815/3683566991","heartbeat_front_addr":"172.21.15.110:6813/3683566991","state":["exists","up"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6801","nonce":2374111370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6803","nonce":2374111370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6807","nonce":2374111370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6805","nonce":2374111370}]},"public_addr":"172.21.15.110:6801/2374111370","cluster_addr":"172.21.15.110:6803/2374111370","heartbeat_back_addr":"172.21.15.110:6807/2374111370","heartbeat_front_addr":"172.21.15.110:6805/2374111370","state":["exists","up"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6827","nonce":1007517023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6829","nonce":1007517023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6833","nonce":1007517023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6831","nonce":1007517023}]},"public_addr":"172.21.15.162:6827/1007517023","cluster_addr":"172.21.15.162:6829/1007517023","heartbeat_back_addr":"172.21.15.162:6833/1007517023","heartbeat_front_addr":"172.21.15.162:6831/1007517023","state":["exists","up"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6803","nonce":946981202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6805","nonce":946981202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6809","nonce":946981202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6807","nonce":946981202}]},"public_addr":"172.21.15.162:6803/946981202","cluster_addr":"172.21.15.162:6805/946981202","heartbeat_back_addr":"172.21.15.162:6809/946981202","heartbeat_front_addr":"172.21.15.162:6807/946981202","state":["exists","up"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6811","nonce":3628034141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6813","nonce":3628034141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6817","nonce":3628034141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6815","nonce":3628034141}]},"public_addr":"172.21.15.162:6811/3628034141","cluster_addr":"172.21.15.162:6813/3628034141","heartbeat_back_addr":"172.21.15.162:6817/3628034141","heartbeat_front_addr":"172.21.15.162:6815/3628034141","state":["exists","up"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6819","nonce":68785612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6821","nonce":68785612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6825","nonce":68785612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6823","nonce":68785612}]},"public_addr":"172.21.15.162:6819/68785612","cluster_addr":"172.21.15.162:6821/68785612","heartbeat_back_addr":"172.21.15.162:6825/68785612","heartbeat_front_addr":"172.21.15.162:6823/68785612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.313498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:10.279187+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.161615+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.904665+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:08.256516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.104354+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:08.722052+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:07.959074+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:43:27.424 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 10 librados: watch_flush enter 2024-01-27T03:43:27.424 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 10 librados: watch_flush exit 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 20 client.4272.objecter shutdown clearing up homeless session... 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 10 client.4272.objecter successfully canceled tick 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 msgr2=0x7fb75806ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb768001560 tx=0x7fb768001a90 comp rx=0 tx=0).stop 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 10 monclient: shutdown 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb778144ba0 msgr2=0x7fb778145020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb778144ba0 0x7fb778145020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb77000c3c0 tx=0x7fb77000c880 comp rx=0 tx=0).stop 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 shutdown_connections 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77813fa10 0x7fb7781d8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb778144ba0 0x7fb778145020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 --2- 172.21.15.110:0/1008197205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7780b4c80 0x7fb7781407b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:27.425 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 >> 172.21.15.110:0/1008197205 conn(0x7fb778071f90 msgr2=0x7fb7780b7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 shutdown_connections 2024-01-27T03:43:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 -- 172.21.15.110:0/1008197205 wait complete. 2024-01-27T03:43:27.426 INFO:teuthology.misc.health.smithi110.stderr:2024-01-27T03:43:27.423+0000 7fb77c890640 1 librados: shutdown 2024-01-27T03:43:27.434 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-27T03:43:27.434 INFO:tasks.ceph:Creating RBD pool 2024-01-27T03:43:27.434 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-27T03:43:27.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.533+0000 7f592c5c7640 10 monclient: get_monmap_and_config 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.533+0000 7f592c5c7640 10 monclient: build_initial_monmap 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.533+0000 7f592c5c7640 10 monclient: monmap: 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:27.534573+0000 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:27.534573+0000 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:27.535 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:27.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.533+0000 7f592c5c7640 1 Processor -- start 2024-01-27T03:43:27.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 1 -- start start 2024-01-27T03:43:27.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 10 monclient: init 2024-01-27T03:43:27.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:27.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-01-27T03:43:27.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f592406bcb0 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406c090 0x7f59241457a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 10 monclient(hunting): picked mon.noname-b con 0x7f592406c090 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5924145ce0 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240ae4b0 con 0x7f592406bcb0 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240ae090 con 0x7f5924145ce0 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592406ca50 con 0x7f592406c090 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.534+0000 7f592c5c7640 10 monclient(hunting): authenticate will time out at 1490.888729s 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.535+0000 7f592ab64640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f5924145ce0 0x7f592414c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:36712/0 (socket says 172.21.15.110:36712) 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.535+0000 7f592ab64640 1 -- 172.21.15.110:0/523782379 learned_addr learned my addr 172.21.15.110:0/523782379 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.535+0000 7f5929b62640 1 --2- 172.21.15.110:0/523782379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406c090 0x7f59241457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.535+0000 7f5929b62640 10 monclient(hunting): get_auth_request con 0x7f592406c090 auth_method 0 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.535+0000 7f5929b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:27.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.535+0000 7f5929b62640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:27.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.535+0000 7f5929b62640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:27.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.536+0000 7f5929b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:27.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.536+0000 7f5929b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:27.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.536+0000 7f5929b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 1 -- 172.21.15.110:0/523782379 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555771895 0 0) 0x7f59240ae090 con 0x7f5924145ce0 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 10 monclient(hunting): _negotiate 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 10 monclient(hunting): my global_id is 4267 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 1 -- 172.21.15.110:0/523782379 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5910003520 con 0x7f5924145ce0 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 1 -- 172.21.15.110:0/523782379 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373580406 0 0) 0x7f59240ae4b0 con 0x7f592406bcb0 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 10 monclient(hunting): _negotiate 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:27.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 10 monclient(hunting): my global_id is 4275 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.537+0000 7f5929361640 1 -- 172.21.15.110:0/523782379 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59240ae090 con 0x7f592406bcb0 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.538+0000 7f5929b62640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.538+0000 7f5929b62640 1 -- 172.21.15.110:0/523782379 >> v1:172.21.15.162:6789/0 conn(0x7f5924145ce0 legacy=0x7f592414c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.538+0000 7f5929b62640 1 -- 172.21.15.110:0/523782379 >> v1:172.21.15.110:6789/0 conn(0x7f592406bcb0 legacy=0x7f592413f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.538+0000 7f5929b62640 10 monclient: _finish_hunting 0 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.538+0000 7f5929b62640 1 monclient: found mon.noname-b 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.538+0000 7f5929b62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.538+0000 7f5929b62640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.538+0000 7f5929b62640 1 -- 172.21.15.110:0/523782379 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59240b0100 con 0x7f592406c090 2024-01-27T03:43:27.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.539+0000 7f5929b62640 1 --2- 172.21.15.110:0/523782379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406c090 0x7f59241457a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f59140097d0 tx=0x7f591402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75bbd980a89b61a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:27.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 1 -- 172.21.15.110:0/523782379 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914031800 con 0x7f592406c090 2024-01-27T03:43:27.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:27.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:27.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 10 monclient: dump: 2024-01-27T03:43:27.544 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:27.544 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 10 monclient: _finish_auth 0 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 10 monclient: _check_auth_tickets 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 1 -- 172.21.15.110:0/523782379 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5914031960 con 0x7f592406c090 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f592c5c7640 5 monclient: authenticate success, global_id 4235 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f592c5c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:27.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f592c5c7640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f592c5c7640 4 set_mon_vals no callback set 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 1 -- 172.21.15.110:0/523782379 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914031cf0 con 0x7f592406c090 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.543+0000 7f5929361640 10 monclient: dump: 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:27.546 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:27.547 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:27.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 10 monclient: shutdown 2024-01-27T03:43:27.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 1 -- 172.21.15.110:0/523782379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406c090 msgr2=0x7f59241457a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 1 --2- 172.21.15.110:0/523782379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406c090 0x7f59241457a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f59140097d0 tx=0x7f591402b8a0 comp rx=0 tx=0).stop 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 1 -- 172.21.15.110:0/523782379 shutdown_connections 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 1 --2- 172.21.15.110:0/523782379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406c090 0x7f59241457a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 1 -- 172.21.15.110:0/523782379 >> 172.21.15.110:0/523782379 conn(0x7f59240b2640 msgr2=0x7f59240b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 1 -- 172.21.15.110:0/523782379 shutdown_connections 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 1 -- 172.21.15.110:0/523782379 wait complete. 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 10 monclient: build_initial_monmap 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.544+0000 7f592c5c7640 10 monclient: monmap: 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:27.551 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 librados: starting msgr at 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 librados: starting objecter 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 Processor -- start 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 -- start start 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 librados: setting wanted keys 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 librados: calling monclient init 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient: init 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:27.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406bcb0 0x7f5924142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient(hunting): picked mon.noname-c con 0x7f592406bcb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f592406c090 0x7f5924141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient(hunting): picked mon.noname-b con 0x7f592406c090 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5924145ce0 0x7f5924134c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient(hunting): picked mon.noname-a con 0x7f5924145ce0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5924140ff0 con 0x7f5924145ce0 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59241421e0 con 0x7f592406c090 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5924141c50 con 0x7f592406bcb0 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.545+0000 7f592c5c7640 10 monclient(hunting): authenticate will time out at 1490.899951s 2024-01-27T03:43:27.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592ab64640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5924145ce0 0x7f5924134c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592a363640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406bcb0 0x7f5924142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f5929b62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f592406c090 0x7f5924141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592ab64640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5924145ce0 0x7f5924134c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33774/0 (socket says 172.21.15.110:33774) 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592ab64640 1 -- 172.21.15.110:0/3789755527 learned_addr learned my addr 172.21.15.110:0/3789755527 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592ab64640 10 monclient(hunting): get_auth_request con 0x7f5924145ce0 auth_method 0 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592ab64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592ab64640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592ab64640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592a363640 10 monclient(hunting): get_auth_request con 0x7f592406bcb0 auth_method 0 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592a363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592a363640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f592a363640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f5929b62640 10 monclient(hunting): get_auth_request con 0x7f592406c090 auth_method 0 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f5929b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f5929b62640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.546+0000 7f5929b62640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.547+0000 7f592ab64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.547+0000 7f592ab64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.547+0000 7f592ab64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.547+0000 7f592a363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.547+0000 7f592a363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.547+0000 7f592a363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.547+0000 7f5929b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.547+0000 7f5929b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.547+0000 7f5929b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 1 -- 172.21.15.110:0/3789755527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406bcb0 msgr2=0x7f5924142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406bcb0 0x7f5924142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 1 -- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f592406c090 msgr2=0x7f5924141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:27.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f592406c090 0x7f5924141710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:27.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 10 monclient: _finish_hunting 0 2024-01-27T03:43:27.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 1 monclient: found mon.noname-a 2024-01-27T03:43:27.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:27.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:27.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f592ab64640 1 -- 172.21.15.110:0/3789755527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5914009400 con 0x7f5924145ce0 2024-01-27T03:43:27.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.549+0000 7f5929b62640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f592406c090 0x7f5924141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T03:43:27.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.551+0000 7f592ab64640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5924145ce0 0x7f5924134c50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f591c00c9f0 tx=0x7f591c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:27.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.551+0000 7f591affd640 10 client.?.objecter ms_handle_connect 0x7f5924145ce0 2024-01-27T03:43:27.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.551+0000 7f591affd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 1 -- 172.21.15.110:0/3789755527 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c00ab70 con 0x7f5924145ce0 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 10 monclient: dump: 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 10 monclient: _finish_auth 0 2024-01-27T03:43:27.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 1 -- 172.21.15.110:0/3789755527 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c00acd0 con 0x7f5924145ce0 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 1 -- 172.21.15.110:0/3789755527 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c012620 con 0x7f5924145ce0 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f592c5c7640 5 monclient: authenticate success, global_id 4278 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f592c5c7640 10 monclient: _renew_subs 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f592c5c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5924141e70 con 0x7f5924145ce0 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f5928b60640 4 set_mon_vals no callback set 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f591affd640 10 monclient: dump: 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:27.559 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f592c5c7640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f592c5c7640 10 monclient: _renew_subs 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f592c5c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5924135370 con 0x7f5924145ce0 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.555+0000 7f592c5c7640 1 librados: init done 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.556+0000 7f592c5c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.556+0000 7f592c5c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:27.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.556+0000 7f592c5c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:27.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.556+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59241d7640 con 0x7f5924145ce0 2024-01-27T03:43:27.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.574+0000 7f591affd640 1 -- 172.21.15.110:0/3789755527 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f591c007070 con 0x7f5924145ce0 2024-01-27T03:43:27.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.574+0000 7f591affd640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f590806c9f0 0x7f590806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:27.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.574+0000 7f592a363640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f590806c9f0 0x7f590806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:27.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.575+0000 7f592a363640 10 monclient: get_auth_request con 0x7f590806c9f0 auth_method 0 2024-01-27T03:43:27.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.575+0000 7f591affd640 10 client.4278.objecter ms_handle_connect 0x7f590806c9f0 2024-01-27T03:43:27.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.575+0000 7f592a363640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f590806c9f0 0x7f590806eeb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5920007fd0 tx=0x7f5920009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:27.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.576+0000 7f591affd640 1 -- 172.21.15.110:0/3789755527 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f591c00ae40 con 0x7f5924145ce0 2024-01-27T03:43:27.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.576+0000 7f591affd640 10 client.4278.objecter ms_dispatch 0x7f5924000b90 osd_map(23..23 src has 1..23) v4 2024-01-27T03:43:27.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.576+0000 7f591affd640 3 client.4278.objecter handle_osd_map got epochs [23,23] > 0 2024-01-27T03:43:27.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.576+0000 7f591affd640 3 client.4278.objecter handle_osd_map decoding full epoch 23 2024-01-27T03:43:27.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.576+0000 7f591affd640 20 client.4278.objecter dump_active .. 0 homeless 2024-01-27T03:43:27.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.809+0000 7f591affd640 1 -- 172.21.15.110:0/3789755527 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f591c0547f0 con 0x7f5924145ce0 2024-01-27T03:43:27.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.809+0000 7f591affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:27.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.809+0000 7f591affd640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.980+0000 7f592c5c7640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-27T03:43:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.980+0000 7f592c5c7640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-27T03:43:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.980+0000 7f592c5c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:27.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:27.980+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f59241d7640 con 0x7f5924145ce0 2024-01-27T03:43:28.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.452+0000 7f591affd640 1 -- 172.21.15.110:0/3789755527 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v24) v1 ==== 109+0+0 (secure 0 0 0) 0x7f59241d7640 con 0x7f5924145ce0 2024-01-27T03:43:28.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.452+0000 7f591affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-27T03:43:28.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.452+0000 7f591affd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-01-27T03:43:28.453 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'rbd' created 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 10 librados: watch_flush enter 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 10 librados: watch_flush exit 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 20 client.4278.objecter shutdown clearing up homeless session... 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 10 client.4278.objecter successfully canceled tick 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f590806c9f0 msgr2=0x7f590806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f590806c9f0 0x7f590806eeb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5920007fd0 tx=0x7f5920009040 comp rx=0 tx=0).stop 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 10 monclient: shutdown 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5924145ce0 msgr2=0x7f5924134c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5924145ce0 0x7f5924134c50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f591c00c9f0 tx=0x7f591c00ceb0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 shutdown_connections 2024-01-27T03:43:28.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f590806c9f0 0x7f590806eeb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5924145ce0 0x7f5924134c50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f592406c090 0x7f5924141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 --2- 172.21.15.110:0/3789755527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f592406bcb0 0x7f5924142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 >> 172.21.15.110:0/3789755527 conn(0x7f59240b2640 msgr2=0x7f5924137a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:28.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 shutdown_connections 2024-01-27T03:43:28.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 -- 172.21.15.110:0/3789755527 wait complete. 2024-01-27T03:43:28.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.454+0000 7f592c5c7640 1 librados: shutdown 2024-01-27T03:43:28.466 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-27T03:43:28.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.572+0000 7f39fbc97640 10 monclient: get_monmap_and_config 2024-01-27T03:43:28.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.572+0000 7f39fbc97640 10 monclient: build_initial_monmap 2024-01-27T03:43:28.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.572+0000 7f39fbc97640 10 monclient: monmap: 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:28.573755+0000 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:28.573755+0000 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:28.574 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.573+0000 7f39fbc97640 1 Processor -- start 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.573+0000 7f39fbc97640 1 -- start start 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.573+0000 7f39fbc97640 10 monclient: init 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.573+0000 7f39fbc97640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.573+0000 7f39fbc97640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f39f406a940 0x7f39f41426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 10 monclient(hunting): picked mon.noname-a con 0x7f39f406a940 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f39f40a9e40 0x7f39f4142c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 10 monclient(hunting): picked mon.noname-c con 0x7f39f40a9e40 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 10 monclient(hunting): picked mon.noname-b con 0x7f39f40b2d40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:28.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:28.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f406ad20 con 0x7f39f406a940 2024-01-27T03:43:28.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:28.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f40aac50 con 0x7f39f40a9e40 2024-01-27T03:43:28.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:28.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f40aaf30 con 0x7f39f40b2d40 2024-01-27T03:43:28.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:28.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.574+0000 7f39fbc97640 10 monclient(hunting): authenticate will time out at 1491.928111s 2024-01-27T03:43:28.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.605+0000 7f39fa234640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:28.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.606+0000 7f39fa234640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57708/0 (socket says 172.21.15.110:57708) 2024-01-27T03:43:28.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.606+0000 7f39fa234640 1 -- 172.21.15.110:0/404964355 learned_addr learned my addr 172.21.15.110:0/404964355 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:28.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.606+0000 7f39fa234640 10 monclient(hunting): get_auth_request con 0x7f39f40b2d40 auth_method 0 2024-01-27T03:43:28.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.606+0000 7f39fa234640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:28.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.606+0000 7f39fa234640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:28.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.606+0000 7f39fa234640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:28.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.606+0000 7f39fa234640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:28.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.606+0000 7f39fa234640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:28.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.606+0000 7f39fa234640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:28.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-01-27T03:43:28.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 1 -- 172.21.15.110:0/404964355 >> v2:172.21.15.162:3300/0 conn(0x7f39f40a9e40 msgr2=0x7f39f4142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:28.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 1 --2- 172.21.15.110:0/404964355 >> v2:172.21.15.162:3300/0 conn(0x7f39f40a9e40 0x7f39f4142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 1 -- 172.21.15.110:0/404964355 >> v2:172.21.15.110:3300/0 conn(0x7f39f406a940 msgr2=0x7f39f41426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:28.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 1 --2- 172.21.15.110:0/404964355 >> v2:172.21.15.110:3300/0 conn(0x7f39f406a940 0x7f39f41426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 10 monclient: _finish_hunting 0 2024-01-27T03:43:28.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 1 monclient: found mon.noname-b 2024-01-27T03:43:28.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:28.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:28.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.608+0000 7f39fa234640 1 -- 172.21.15.110:0/404964355 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f41421b0 con 0x7f39f40b2d40 2024-01-27T03:43:28.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.610+0000 7f39fa234640 1 --2- 172.21.15.110:0/404964355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40b3150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f39f4072680 tx=0x7f39ec02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c4f79493a0abef9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 1 -- 172.21.15.110:0/404964355 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec02ca70 con 0x7f39f40b2d40 2024-01-27T03:43:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 10 monclient: dump: 2024-01-27T03:43:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 10 monclient: _finish_auth 0 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 10 monclient: _check_auth_tickets 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 1 -- 172.21.15.110:0/404964355 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec02cbd0 con 0x7f39f40b2d40 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 1 -- 172.21.15.110:0/404964355 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec0343f0 con 0x7f39f40b2d40 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39fbc97640 5 monclient: authenticate success, global_id 4241 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39fbc97640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:28.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39fbc97640 4 set_mon_vals no callback set 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.614+0000 7f39f8a31640 10 monclient: dump: 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:28.617 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:28.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 10 monclient: shutdown 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 1 -- 172.21.15.110:0/404964355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 msgr2=0x7f39f40b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 1 --2- 172.21.15.110:0/404964355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40b3150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f39f4072680 tx=0x7f39ec02bed0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 1 -- 172.21.15.110:0/404964355 shutdown_connections 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 1 --2- 172.21.15.110:0/404964355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40b3150 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 1 --2- 172.21.15.110:0/404964355 >> v2:172.21.15.162:3300/0 conn(0x7f39f40a9e40 0x7f39f4142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 1 --2- 172.21.15.110:0/404964355 >> v2:172.21.15.110:3300/0 conn(0x7f39f406a940 0x7f39f41426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 1 -- 172.21.15.110:0/404964355 >> 172.21.15.110:0/404964355 conn(0x7f39f40b7d70 msgr2=0x7f39f40b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 1 -- 172.21.15.110:0/404964355 shutdown_connections 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.615+0000 7f39fbc97640 1 -- 172.21.15.110:0/404964355 wait complete. 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.616+0000 7f39fbc97640 10 monclient: build_initial_monmap 2024-01-27T03:43:28.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.616+0000 7f39fbc97640 10 monclient: monmap: 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.616+0000 7f39fbc97640 1 librados: starting msgr at 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.616+0000 7f39fbc97640 1 librados: starting objecter 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.616+0000 7f39fbc97640 1 Processor -- start 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.616+0000 7f39fbc97640 1 -- start start 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.616+0000 7f39fbc97640 1 librados: setting wanted keys 2024-01-27T03:43:28.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.616+0000 7f39fbc97640 1 librados: calling monclient init 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.616+0000 7f39fbc97640 10 monclient: init 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39f406a940 0x7f39f40add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient(hunting): picked mon.noname-b con 0x7f39f406a940 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f40a9e40 0x7f39f41dac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient(hunting): picked mon.noname-a con 0x7f39f40a9e40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient(hunting): picked mon.noname-c con 0x7f39f40b2d40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f41d4cd0 con 0x7f39f40a9e40 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f41da750 con 0x7f39f406a940 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f40b0580 con 0x7f39f40b2d40 2024-01-27T03:43:28.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fbc97640 10 monclient(hunting): authenticate will time out at 1491.971093s 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39f9a33640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39f406a940 0x7f39f40add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39fa234640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39f9a33640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39f406a940 0x7f39f40add00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:46484/0 (socket says 172.21.15.110:46484) 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39f9a33640 1 -- 172.21.15.110:0/2489475158 learned_addr learned my addr 172.21.15.110:0/2489475158 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39f9a33640 10 monclient(hunting): get_auth_request con 0x7f39f406a940 auth_method 0 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39f9a33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39f9a33640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.617+0000 7f39f9a33640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39fa234640 10 monclient(hunting): get_auth_request con 0x7f39f40b2d40 auth_method 0 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39fa234640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39fa234640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39fa234640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39f9a33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39f9a33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39f9a33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39fa234640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39fa234640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.618+0000 7f39fa234640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 1 -- 172.21.15.110:0/2489475158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 msgr2=0x7f39f40ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40ae810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 1 -- 172.21.15.110:0/2489475158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f40a9e40 msgr2=0x7f39f41dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f40a9e40 0x7f39f41dac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 10 monclient: _finish_hunting 0 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 1 monclient: found mon.noname-b 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.621+0000 7f39f9a33640 1 -- 172.21.15.110:0/2489475158 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec009b30 con 0x7f39f406a940 2024-01-27T03:43:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.623+0000 7f39f9a33640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39f406a940 0x7f39f40add00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f39f000b4a0 tx=0x7f39f000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.623+0000 7f39ea7fc640 10 client.?.objecter ms_handle_connect 0x7f39f406a940 2024-01-27T03:43:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.623+0000 7f39ea7fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 1 -- 172.21.15.110:0/2489475158 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0010610 con 0x7f39f406a940 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 10 monclient: dump: 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 1 -- 172.21.15.110:0/2489475158 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f0010770 con 0x7f39f406a940 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39fbc97640 5 monclient: authenticate success, global_id 4273 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39fbc97640 10 monclient: _renew_subs 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39fbc97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f40b0800 con 0x7f39f406a940 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 1 -- 172.21.15.110:0/2489475158 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0010b20 con 0x7f39f406a940 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ea7fc640 10 monclient: dump: 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.628+0000 7f39fbc97640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.628+0000 7f39fbc97640 10 monclient: _renew_subs 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.628+0000 7f39fbc97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:28.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.628+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f41db390 con 0x7f39f406a940 2024-01-27T03:43:28.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.627+0000 7f39ebfff640 4 set_mon_vals no callback set 2024-01-27T03:43:28.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.628+0000 7f39fbc97640 1 librados: init done 2024-01-27T03:43:28.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.628+0000 7f39fbc97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:28.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.628+0000 7f39fbc97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:28.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.628+0000 7f39fbc97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:28.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.628+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f4074d10 con 0x7f39f406a940 2024-01-27T03:43:28.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.888+0000 7f39ea7fc640 1 -- 172.21.15.110:0/2489475158 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f39f0010c80 con 0x7f39f406a940 2024-01-27T03:43:28.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.889+0000 7f39ea7fc640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f39d806c9a0 0x7f39d806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:28.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.889+0000 7f39f9232640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f39d806c9a0 0x7f39d806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:28.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.889+0000 7f39f9232640 10 monclient: get_auth_request con 0x7f39d806c9a0 auth_method 0 2024-01-27T03:43:28.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.889+0000 7f39f9232640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f39d806c9a0 0x7f39d806ee60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f39f4146410 tx=0x7f39e4006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:28.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.889+0000 7f39ea7fc640 10 client.4273.objecter ms_handle_connect 0x7f39d806c9a0 2024-01-27T03:43:28.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.891+0000 7f39ea7fc640 1 -- 172.21.15.110:0/2489475158 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f39f0058410 con 0x7f39f406a940 2024-01-27T03:43:28.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.891+0000 7f39ea7fc640 10 client.4273.objecter ms_dispatch 0x7f39f4000b90 osd_map(24..24 src has 1..24) v4 2024-01-27T03:43:28.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.891+0000 7f39ea7fc640 3 client.4273.objecter handle_osd_map got epochs [24,24] > 0 2024-01-27T03:43:28.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.891+0000 7f39ea7fc640 3 client.4273.objecter handle_osd_map decoding full epoch 24 2024-01-27T03:43:28.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.891+0000 7f39ea7fc640 20 client.4273.objecter dump_active .. 0 homeless 2024-01-27T03:43:28.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.901+0000 7f39ea7fc640 1 -- 172.21.15.110:0/2489475158 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f39f00546b0 con 0x7f39f406a940 2024-01-27T03:43:28.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.901+0000 7f39ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:28.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:28.901+0000 7f39ea7fc640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:29.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.123+0000 7f39fbc97640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-27T03:43:29.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.123+0000 7f39fbc97640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-27T03:43:29.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.123+0000 7f39fbc97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:29.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.123+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f39f404eac0 con 0x7f39f406a940 2024-01-27T03:43:29.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.750+0000 7f39ea7fc640 1 -- 172.21.15.110:0/2489475158 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v25) v1 ==== 173+0+0 (secure 0 0 0) 0x7f39f001e020 con 0x7f39f406a940 2024-01-27T03:43:29.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.750+0000 7f39ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-27T03:43:29.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.750+0000 7f39ea7fc640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-01-27T03:43:29.752 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-27T03:43:29.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 10 librados: watch_flush enter 2024-01-27T03:43:29.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 10 librados: watch_flush exit 2024-01-27T03:43:29.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 20 client.4273.objecter shutdown clearing up homeless session... 2024-01-27T03:43:29.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 10 client.4273.objecter successfully canceled tick 2024-01-27T03:43:29.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f39d806c9a0 msgr2=0x7f39d806ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f39d806c9a0 0x7f39d806ee60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f39f4146410 tx=0x7f39e4006c90 comp rx=0 tx=0).stop 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 10 monclient: shutdown 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39f406a940 msgr2=0x7f39f40add00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.752+0000 7f39fbc97640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39f406a940 0x7f39f40add00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f39f000b4a0 tx=0x7f39f000b960 comp rx=0 tx=0).stop 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.753+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 shutdown_connections 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.753+0000 7f39fbc97640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f39d806c9a0 0x7f39d806ee60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.753+0000 7f39fbc97640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39f40b2d40 0x7f39f40ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.753+0000 7f39fbc97640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f40a9e40 0x7f39f41dac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.753+0000 7f39fbc97640 1 --2- 172.21.15.110:0/2489475158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39f406a940 0x7f39f40add00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.753+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 >> 172.21.15.110:0/2489475158 conn(0x7f39f40b7d70 msgr2=0x7f39f4140ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:29.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.753+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 shutdown_connections 2024-01-27T03:43:29.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.753+0000 7f39fbc97640 1 -- 172.21.15.110:0/2489475158 wait complete. 2024-01-27T03:43:29.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.753+0000 7f39fbc97640 1 librados: shutdown 2024-01-27T03:43:29.767 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-27T03:43:29.767 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-27T03:43:29.768 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-27T03:43:29.768 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-27T03:43:29.768 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-27T03:43:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.877+0000 7f69d4421640 10 monclient: get_monmap_and_config 2024-01-27T03:43:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.877+0000 7f69d4421640 10 monclient: build_initial_monmap 2024-01-27T03:43:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.877+0000 7f69d4421640 10 monclient: monmap: 2024-01-27T03:43:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:29.878832+0000 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:29.878832+0000 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:29.879 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:29.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.878+0000 7f69d4421640 1 Processor -- start 2024-01-27T03:43:29.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.878+0000 7f69d4421640 1 -- start start 2024-01-27T03:43:29.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.878+0000 7f69d4421640 10 monclient: init 2024-01-27T03:43:29.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.878+0000 7f69d4421640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:29.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.878+0000 7f69d4421640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-01-27T03:43:29.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.878+0000 7f69d4421640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc0b4680 0x7f69cc142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 10 monclient(hunting): picked mon.noname-b con 0x7f69cc0b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69cc0b4a60 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f69cc142eb0 0x7f69cc0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 10 monclient(hunting): picked mon.noname-c con 0x7f69cc142eb0 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc0afbe0 con 0x7f69cc0b4a60 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69cc142240 con 0x7f69cc142eb0 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0b0890 con 0x7f69cc0b4680 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d4421640 10 monclient(hunting): authenticate will time out at 1493.233024s 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d21bd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc0b4680 0x7f69cc142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d19bc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f69cc0b4a60 0x7f69cc0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:49082/0 (socket says 172.21.15.110:49082) 2024-01-27T03:43:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.879+0000 7f69d19bc640 1 -- 172.21.15.110:0/1132317467 learned_addr learned my addr 172.21.15.110:0/1132317467 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d29be640 1 --2- 172.21.15.110:0/1132317467 >> v2:172.21.15.162:3300/0 conn(0x7f69cc142eb0 0x7f69cc0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d21bd640 10 monclient(hunting): get_auth_request con 0x7f69cc0b4680 auth_method 0 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d21bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d21bd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d21bd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d29be640 10 monclient(hunting): get_auth_request con 0x7f69cc142eb0 auth_method 0 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d29be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d29be640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d29be640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d21bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d21bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.880+0000 7f69d21bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.881+0000 7f69d29be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.881+0000 7f69d29be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.881+0000 7f69d29be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.882+0000 7f69d21bd640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-01-27T03:43:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.882+0000 7f69d21bd640 1 -- 172.21.15.110:0/1132317467 >> v2:172.21.15.162:3300/0 conn(0x7f69cc142eb0 msgr2=0x7f69cc0b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.882+0000 7f69d21bd640 1 --2- 172.21.15.110:0/1132317467 >> v2:172.21.15.162:3300/0 conn(0x7f69cc142eb0 0x7f69cc0b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:29.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.882+0000 7f69d21bd640 1 -- 172.21.15.110:0/1132317467 >> v1:172.21.15.162:6789/0 conn(0x7f69cc0b4a60 legacy=0x7f69cc0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.882+0000 7f69d21bd640 10 monclient: _finish_hunting 0 2024-01-27T03:43:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.882+0000 7f69d21bd640 1 monclient: found mon.noname-b 2024-01-27T03:43:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.882+0000 7f69d21bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.882+0000 7f69d21bd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.882+0000 7f69d21bd640 1 -- 172.21.15.110:0/1132317467 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc0b0b00 con 0x7f69cc0b4680 2024-01-27T03:43:29.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.884+0000 7f69d21bd640 1 --2- 172.21.15.110:0/1132317467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc0b4680 0x7f69cc142970 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f69c80097a0 tx=0x7f69c802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eabfc38bfc82ed66 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 1 -- 172.21.15.110:0/1132317467 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c802e070 con 0x7f69cc0b4680 2024-01-27T03:43:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 10 monclient: dump: 2024-01-27T03:43:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:29.888 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 10 monclient: _finish_auth 0 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 10 monclient: _check_auth_tickets 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 1 -- 172.21.15.110:0/1132317467 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c80315a0 con 0x7f69cc0b4680 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d4421640 5 monclient: authenticate success, global_id 4247 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d4421640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:29.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d4421640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 1 -- 172.21.15.110:0/1132317467 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8031930 con 0x7f69cc0b4680 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d4421640 4 set_mon_vals no callback set 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.887+0000 7f69d11bb640 10 monclient: dump: 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:29.890 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 10 monclient: shutdown 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 1 -- 172.21.15.110:0/1132317467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc0b4680 msgr2=0x7f69cc142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 1 --2- 172.21.15.110:0/1132317467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc0b4680 0x7f69cc142970 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f69c80097a0 tx=0x7f69c802b840 comp rx=0 tx=0).stop 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 1 -- 172.21.15.110:0/1132317467 shutdown_connections 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 1 --2- 172.21.15.110:0/1132317467 >> v2:172.21.15.162:3300/0 conn(0x7f69cc142eb0 0x7f69cc0b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 1 --2- 172.21.15.110:0/1132317467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc0b4680 0x7f69cc142970 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 1 -- 172.21.15.110:0/1132317467 >> 172.21.15.110:0/1132317467 conn(0x7f69cc066490 msgr2=0x7f69cc0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 1 -- 172.21.15.110:0/1132317467 shutdown_connections 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.888+0000 7f69d4421640 1 -- 172.21.15.110:0/1132317467 wait complete. 2024-01-27T03:43:29.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.889+0000 7f69d4421640 10 monclient: build_initial_monmap 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.889+0000 7f69d4421640 10 monclient: monmap: 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.889+0000 7f69d4421640 1 librados: starting msgr at 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.889+0000 7f69d4421640 1 librados: starting objecter 2024-01-27T03:43:29.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.889+0000 7f69d4421640 1 Processor -- start 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.889+0000 7f69d4421640 1 -- start start 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.889+0000 7f69d4421640 1 librados: setting wanted keys 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.889+0000 7f69d4421640 1 librados: calling monclient init 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.889+0000 7f69d4421640 10 monclient: init 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69cc0b4680 0x7f69cc145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient(hunting): picked mon.noname-a con 0x7f69cc0b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69cc142eb0 0x7f69cc13c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient(hunting): picked mon.noname-b con 0x7f69cc142eb0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc13d7a0 0x7f69cc13e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient(hunting): picked mon.noname-c con 0x7f69cc13d7a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:29.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc13e1b0 con 0x7f69cc0b4680 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0a9f60 con 0x7f69cc142eb0 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc13dc50 con 0x7f69cc13d7a0 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d4421640 10 monclient(hunting): authenticate will time out at 1493.244259s 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d19bc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69cc142eb0 0x7f69cc13c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d21bd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69cc0b4680 0x7f69cc145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.890+0000 7f69d29be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc13d7a0 0x7f69cc13e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d19bc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69cc142eb0 0x7f69cc13c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40506/0 (socket says 172.21.15.110:40506) 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d19bc640 1 -- 172.21.15.110:0/2459983475 learned_addr learned my addr 172.21.15.110:0/2459983475 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d19bc640 10 monclient(hunting): get_auth_request con 0x7f69cc142eb0 auth_method 0 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d19bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d19bc640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d19bc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:29.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d21bd640 10 monclient(hunting): get_auth_request con 0x7f69cc0b4680 auth_method 0 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d21bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d21bd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d21bd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d29be640 10 monclient(hunting): get_auth_request con 0x7f69cc13d7a0 auth_method 0 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d29be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d29be640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d29be640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d19bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d19bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d19bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d21bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d21bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d21bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d29be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:29.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d29be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:29.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.891+0000 7f69d29be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:29.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 10 monclient(hunting): handle_auth_done global_id 4279 payload 824 2024-01-27T03:43:29.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 1 -- 172.21.15.110:0/2459983475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc13d7a0 msgr2=0x7f69cc13e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:29.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc13d7a0 0x7f69cc13e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:29.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 1 -- 172.21.15.110:0/2459983475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69cc0b4680 msgr2=0x7f69cc145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:29.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69cc0b4680 0x7f69cc145800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:29.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 10 monclient: _finish_hunting 0 2024-01-27T03:43:29.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 1 monclient: found mon.noname-b 2024-01-27T03:43:29.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:29.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.894+0000 7f69d19bc640 1 -- 172.21.15.110:0/2459983475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8009400 con 0x7f69cc142eb0 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.896+0000 7f69d19bc640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69cc142eb0 0x7f69cc13c060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00c440 tx=0x7f69bc001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.896+0000 7f69c2ffd640 10 client.?.objecter ms_handle_connect 0x7f69cc142eb0 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.896+0000 7f69c2ffd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69c2ffd640 1 -- 172.21.15.110:0/2459983475 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc005ce0 con 0x7f69cc142eb0 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69c2ffd640 10 monclient: dump: 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:29.901 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69c2ffd640 10 monclient: _finish_auth 0 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69c2ffd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69d4421640 5 monclient: authenticate success, global_id 4279 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69d4421640 10 monclient: _renew_subs 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69d4421640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69cc13eb80 con 0x7f69cc142eb0 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69d4421640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69d4421640 10 monclient: _renew_subs 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69d4421640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69cc1dae10 con 0x7f69cc142eb0 2024-01-27T03:43:29.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.899+0000 7f69d4421640 1 librados: init done 2024-01-27T03:43:29.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69d4421640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:29.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69d4421640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:29.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69d4421640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:29.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69cc074d10 con 0x7f69cc142eb0 2024-01-27T03:43:29.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69c2ffd640 1 -- 172.21.15.110:0/2459983475 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc001120 con 0x7f69cc142eb0 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69c2ffd640 1 -- 172.21.15.110:0/2459983475 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc004a20 con 0x7f69cc142eb0 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69d09ba640 4 set_mon_vals no callback set 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:29.900+0000 7f69c2ffd640 10 monclient: dump: 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:29.904 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:29.905 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:29.905 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:30.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.211+0000 7f69c2ffd640 1 -- 172.21.15.110:0/2459983475 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f69bc007070 con 0x7f69cc142eb0 2024-01-27T03:43:30.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.211+0000 7f69c2ffd640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f69b006c9d0 0x7f69b006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:30.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.212+0000 7f69d21bd640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f69b006c9d0 0x7f69b006ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:30.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.212+0000 7f69d21bd640 10 monclient: get_auth_request con 0x7f69b006c9d0 auth_method 0 2024-01-27T03:43:30.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.212+0000 7f69d21bd640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f69b006c9d0 0x7f69b006ee90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:30.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.212+0000 7f69c2ffd640 10 client.4279.objecter ms_handle_connect 0x7f69b006c9d0 2024-01-27T03:43:30.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.217+0000 7f69c2ffd640 1 -- 172.21.15.110:0/2459983475 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f69bc057d80 con 0x7f69cc142eb0 2024-01-27T03:43:30.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.217+0000 7f69c2ffd640 10 client.4279.objecter ms_dispatch 0x7f69cc000b90 osd_map(25..25 src has 1..25) v4 2024-01-27T03:43:30.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.217+0000 7f69c2ffd640 3 client.4279.objecter handle_osd_map got epochs [25,25] > 0 2024-01-27T03:43:30.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.217+0000 7f69c2ffd640 3 client.4279.objecter handle_osd_map decoding full epoch 25 2024-01-27T03:43:30.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.217+0000 7f69c2ffd640 20 client.4279.objecter dump_active .. 0 homeless 2024-01-27T03:43:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.231+0000 7f69c2ffd640 1 -- 172.21.15.110:0/2459983475 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f69bc054020 con 0x7f69cc142eb0 2024-01-27T03:43:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.232+0000 7f69c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.232+0000 7f69c2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.509+0000 7f69d4421640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-01-27T03:43:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.510+0000 7f69d4421640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-01-27T03:43:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.510+0000 7f69d4421640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.510+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f69cc13e320 con 0x7f69cc142eb0 2024-01-27T03:43:30.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.660+0000 7f69c2ffd640 1 -- 172.21.15.110:0/2459983475 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170824 (secure 0 0 0) 0x7f69bc017020 con 0x7f69cc142eb0 2024-01-27T03:43:30.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.660+0000 7f69c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-01-27T03:43:30.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.660+0000 7f69c2ffd640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:30.664 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:43:30.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 10 librados: watch_flush enter 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 10 librados: watch_flush exit 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 20 client.4279.objecter shutdown clearing up homeless session... 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 10 client.4279.objecter successfully canceled tick 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f69b006c9d0 msgr2=0x7f69b006ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f69b006c9d0 0x7f69b006ee90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c803d040 comp rx=0 tx=0).stop 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 10 monclient: shutdown 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69cc142eb0 msgr2=0x7f69cc13c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.662+0000 7f69d4421640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69cc142eb0 0x7f69cc13c060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00c440 tx=0x7f69bc001e70 comp rx=0 tx=0).stop 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.663+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 shutdown_connections 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.663+0000 7f69d4421640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f69b006c9d0 0x7f69b006ee90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.663+0000 7f69d4421640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69cc13d7a0 0x7f69cc13e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.663+0000 7f69d4421640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69cc142eb0 0x7f69cc13c060 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:30.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.663+0000 7f69d4421640 1 --2- 172.21.15.110:0/2459983475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69cc0b4680 0x7f69cc145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:30.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.663+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 >> 172.21.15.110:0/2459983475 conn(0x7f69cc066490 msgr2=0x7f69cc0b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:30.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.663+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 shutdown_connections 2024-01-27T03:43:30.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.663+0000 7f69d4421640 1 -- 172.21.15.110:0/2459983475 wait complete. 2024-01-27T03:43:30.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.663+0000 7f69d4421640 1 librados: shutdown 2024-01-27T03:43:30.675 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3672294946},{"type":"v1","addr":"172.21.15.162:6801","nonce":3672294946}]},"active_addr":"172.21.15.162:6801/3672294946","active_change":"2024-01-27T03:42:30.564410+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":4102,"name":"y","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":2212037868}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":656564673}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":2968373991}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":608766952}]}]} 2024-01-27T03:43:30.681 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-27T03:43:30.681 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-27T03:43:30.681 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T03:43:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.785+0000 7fb0324d4640 10 monclient: get_monmap_and_config 2024-01-27T03:43:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.785+0000 7fb0324d4640 10 monclient: build_initial_monmap 2024-01-27T03:43:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.785+0000 7fb0324d4640 10 monclient: monmap: 2024-01-27T03:43:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:30.786416+0000 2024-01-27T03:43:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:30.786416+0000 2024-01-27T03:43:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:30.787 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:30.787 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:30.787 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:30.787 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:30.787 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:30.787 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.786+0000 7fb0324d4640 1 Processor -- start 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.786+0000 7fb0324d4640 1 -- start start 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.786+0000 7fb0324d4640 10 monclient: init 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.786+0000 7fb0324d4640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.786+0000 7fb0324d4640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.786+0000 7fb0324d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb02c0b4680 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.786+0000 7fb0324d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb02c0b4a60 0x7fb02c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.786+0000 7fb0324d4640 10 monclient(hunting): picked mon.noname-b con 0x7fb02c0b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb0324d4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb02c0b3b30 0x7fb02c0b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb0324d4640 10 monclient(hunting): picked mon.noname-c con 0x7fb02c0b3b30 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb0324d4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02c0afbe0 con 0x7fb02c0b4680 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb0324d4640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:30.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb0324d4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c142240 con 0x7fb02c0b3b30 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb0324d4640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb0324d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0b0890 con 0x7fb02c0b4a60 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb0324d4640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb0324d4640 10 monclient(hunting): authenticate will time out at 1494.141033s 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb030a71640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb02c0b3b30 0x7fb02c0b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb02bfff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb02c0b4680 0x7fb02c142970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:49088/0 (socket says 172.21.15.110:49088) 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.787+0000 7fb02bfff640 1 -- 172.21.15.110:0/821888322 learned_addr learned my addr 172.21.15.110:0/821888322 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.788+0000 7fb030a71640 10 monclient(hunting): get_auth_request con 0x7fb02c0b3b30 auth_method 0 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.788+0000 7fb030a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.788+0000 7fb030a71640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:30.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.788+0000 7fb030a71640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:30.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.788+0000 7fb030a71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:30.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.788+0000 7fb030a71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:30.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.788+0000 7fb030a71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:30.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.791+0000 7fb030a71640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2024-01-27T03:43:30.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.791+0000 7fb030a71640 1 -- 172.21.15.110:0/821888322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb02c0b4a60 msgr2=0x7fb02c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:30.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.791+0000 7fb030a71640 1 --2- 172.21.15.110:0/821888322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb02c0b4a60 0x7fb02c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:30.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.791+0000 7fb030a71640 1 -- 172.21.15.110:0/821888322 >> v1:172.21.15.162:6789/0 conn(0x7fb02c0b4680 legacy=0x7fb02c142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:30.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.791+0000 7fb030a71640 10 monclient: _finish_hunting 0 2024-01-27T03:43:30.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.791+0000 7fb030a71640 1 monclient: found mon.noname-c 2024-01-27T03:43:30.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.791+0000 7fb030a71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:30.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.791+0000 7fb030a71640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:30.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.791+0000 7fb030a71640 1 -- 172.21.15.110:0/821888322 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c0b0b00 con 0x7fb02c0b3b30 2024-01-27T03:43:30.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.793+0000 7fb030a71640 1 --2- 172.21.15.110:0/821888322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb02c0b3b30 0x7fb02c0b19b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb024009d90 tx=0x7fb02402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f033195c14c3311d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:30.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 1 -- 172.21.15.110:0/821888322 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02402cbc0 con 0x7fb02c0b3b30 2024-01-27T03:43:30.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:30.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:30.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 10 monclient: dump: 2024-01-27T03:43:30.799 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 10 monclient: _finish_auth 0 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 10 monclient: _check_auth_tickets 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 1 -- 172.21.15.110:0/821888322 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02402cd20 con 0x7fb02c0b3b30 2024-01-27T03:43:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 1 -- 172.21.15.110:0/821888322 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024034660 con 0x7fb02c0b3b30 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb0324d4640 5 monclient: authenticate success, global_id 4282 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb0324d4640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb0324d4640 4 set_mon_vals no callback set 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.798+0000 7fb02b7fe640 10 monclient: dump: 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:30.802 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:30.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 10 monclient: shutdown 2024-01-27T03:43:30.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:30.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 1 -- 172.21.15.110:0/821888322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb02c0b3b30 msgr2=0x7fb02c0b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:30.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 1 --2- 172.21.15.110:0/821888322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb02c0b3b30 0x7fb02c0b19b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb024009d90 tx=0x7fb02402be10 comp rx=0 tx=0).stop 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 1 -- 172.21.15.110:0/821888322 shutdown_connections 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 1 --2- 172.21.15.110:0/821888322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb02c0b4a60 0x7fb02c0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 1 --2- 172.21.15.110:0/821888322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb02c0b3b30 0x7fb02c0b19b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 1 -- 172.21.15.110:0/821888322 >> 172.21.15.110:0/821888322 conn(0x7fb02c066490 msgr2=0x7fb02c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 1 -- 172.21.15.110:0/821888322 shutdown_connections 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.799+0000 7fb0324d4640 1 -- 172.21.15.110:0/821888322 wait complete. 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.800+0000 7fb0324d4640 10 monclient: build_initial_monmap 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.800+0000 7fb0324d4640 10 monclient: monmap: 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.800+0000 7fb0324d4640 1 librados: starting msgr at 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.800+0000 7fb0324d4640 1 librados: starting objecter 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.800+0000 7fb0324d4640 1 Processor -- start 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.800+0000 7fb0324d4640 1 -- start start 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.800+0000 7fb0324d4640 1 librados: setting wanted keys 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.800+0000 7fb0324d4640 1 librados: calling monclient init 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.800+0000 7fb0324d4640 10 monclient: init 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb02c0b3b30 0x7fb02c13ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient(hunting): picked mon.noname-a con 0x7fb02c0b3b30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb02c0b4680 0x7fb02c0aa900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient(hunting): picked mon.noname-b con 0x7fb02c0b4680 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb02c0a94b0 0x7fb02c0ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient(hunting): picked mon.noname-c con 0x7fb02c0a94b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0ad590 con 0x7fb02c0b3b30 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0aae40 con 0x7fb02c0b4680 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0aca10 con 0x7fb02c0a94b0 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:30.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.801+0000 7fb0324d4640 10 monclient(hunting): authenticate will time out at 1494.155641s 2024-01-27T03:43:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.804+0000 7fb02bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb02c0b3b30 0x7fb02c13ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.805+0000 7fb02bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb02c0b3b30 0x7fb02c13ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:49936/0 (socket says 172.21.15.110:49936) 2024-01-27T03:43:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.805+0000 7fb02bfff640 1 -- 172.21.15.110:0/3430690811 learned_addr learned my addr 172.21.15.110:0/3430690811 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.805+0000 7fb02bfff640 10 monclient(hunting): get_auth_request con 0x7fb02c0b3b30 auth_method 0 2024-01-27T03:43:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.805+0000 7fb02bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.805+0000 7fb02bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.805+0000 7fb02bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.805+0000 7fb02bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:30.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.805+0000 7fb02bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:30.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.806+0000 7fb02bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:30.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.807+0000 7fb023fff640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb02c0b4680 0x7fb02c0aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:30.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.808+0000 7fb030a71640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb02c0a94b0 0x7fb02c0ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.808+0000 7fb023fff640 10 monclient(hunting): get_auth_request con 0x7fb02c0b4680 auth_method 0 2024-01-27T03:43:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.808+0000 7fb023fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.808+0000 7fb023fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.808+0000 7fb023fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb030a71640 10 monclient(hunting): get_auth_request con 0x7fb02c0a94b0 auth_method 0 2024-01-27T03:43:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb030a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb030a71640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb030a71640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb023fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb023fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb023fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:30.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb030a71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:30.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb030a71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:30.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.809+0000 7fb030a71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:30.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-01-27T03:43:30.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 1 -- 172.21.15.110:0/3430690811 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb02c0a94b0 msgr2=0x7fb02c0ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:30.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb02c0a94b0 0x7fb02c0ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 1 -- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb02c0b4680 msgr2=0x7fb02c0aa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb02c0b4680 0x7fb02c0aa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 1 monclient: found mon.noname-a 2024-01-27T03:43:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.810+0000 7fb02bfff640 1 -- 172.21.15.110:0/3430690811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb024009a70 con 0x7fb02c0b3b30 2024-01-27T03:43:30.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.882+0000 7fb02bfff640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb02c0b3b30 0x7fb02c13ab00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00c4a0 tx=0x7fb01c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:30.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.882+0000 7fb0297fa640 10 client.?.objecter ms_handle_connect 0x7fb02c0b3b30 2024-01-27T03:43:30.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.882+0000 7fb0297fa640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 1 -- 172.21.15.110:0/3430690811 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c005ce0 con 0x7fb02c0b3b30 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 10 monclient: dump: 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 10 monclient: _finish_auth 0 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 10 monclient: _check_auth_tickets 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 1 -- 172.21.15.110:0/3430690811 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c005e40 con 0x7fb02c0b3b30 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0324d4640 5 monclient: authenticate success, global_id 4284 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0324d4640 10 monclient: _renew_subs 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0324d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02c0accf0 con 0x7fb02c0b3b30 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 1 -- 172.21.15.110:0/3430690811 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c007270 con 0x7fb02c0b3b30 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:30.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0297fa640 10 monclient: dump: 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb02affd640 4 set_mon_vals no callback set 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0324d4640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0324d4640 10 monclient: _renew_subs 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0324d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c0ae190 con 0x7fb02c0b3b30 2024-01-27T03:43:30.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.890+0000 7fb0324d4640 1 librados: init done 2024-01-27T03:43:30.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.891+0000 7fb0324d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:30.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.891+0000 7fb0324d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:30.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.891+0000 7fb0324d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:30.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:30.891+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c0ad3f0 con 0x7fb02c0b3b30 2024-01-27T03:43:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.132+0000 7fb0297fa640 1 -- 172.21.15.110:0/3430690811 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fb01c007490 con 0x7fb02c0b3b30 2024-01-27T03:43:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.132+0000 7fb0297fa640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb00c06c9a0 0x7fb00c06ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.132+0000 7fb023fff640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb00c06c9a0 0x7fb00c06ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:31.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.132+0000 7fb023fff640 10 monclient: get_auth_request con 0x7fb00c06c9a0 auth_method 0 2024-01-27T03:43:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.133+0000 7fb0297fa640 10 client.4284.objecter ms_handle_connect 0x7fb00c06c9a0 2024-01-27T03:43:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.133+0000 7fb023fff640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb00c06c9a0 0x7fb00c06ee60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0af1a0 tx=0x7fb014009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.145+0000 7fb0297fa640 1 -- 172.21.15.110:0/3430690811 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb01c057b90 con 0x7fb02c0b3b30 2024-01-27T03:43:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.145+0000 7fb0297fa640 10 client.4284.objecter ms_dispatch 0x7fb02c000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.145+0000 7fb0297fa640 3 client.4284.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.145+0000 7fb0297fa640 3 client.4284.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.146+0000 7fb0297fa640 20 client.4284.objecter dump_active .. 0 homeless 2024-01-27T03:43:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.146+0000 7fb0297fa640 1 -- 172.21.15.110:0/3430690811 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fb01c053e30 con 0x7fb02c0b3b30 2024-01-27T03:43:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.146+0000 7fb0297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:31.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.146+0000 7fb0297fa640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:31.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.317+0000 7fb0324d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:31.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.317+0000 7fb0324d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:31.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.317+0000 7fb0324d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:31.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.317+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb02c0ad3f0 con 0x7fb02c0b3b30 2024-01-27T03:43:31.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.350+0000 7fb0297fa640 1 -- 172.21.15.110:0/3430690811 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14675 (secure 0 0 0) 0x7fb02c0ad3f0 con 0x7fb02c0b3b30 2024-01-27T03:43:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.351+0000 7fb0297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:31.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.351+0000 7fb0297fa640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:31.352 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:43:31.352 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":26,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:43:30.682666+0000","last_up_change":"2024-01-27T03:43:21.487334+0000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-27T03:43:13.074487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-27T03:43:27.987244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6817","nonce":1545415171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6819","nonce":1545415171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6824","nonce":1545415171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6821","nonce":1545415171}]},"public_addr":"172.21.15.110:6817/1545415171","cluster_addr":"172.21.15.110:6819/1545415171","heartbeat_back_addr":"172.21.15.110:6824/1545415171","heartbeat_front_addr":"172.21.15.110:6821/1545415171","state":["exists","up"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6823","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6825","nonce":518896551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6827","nonce":518896551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6831","nonce":518896551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6829","nonce":518896551}]},"public_addr":"172.21.15.110:6825/518896551","cluster_addr":"172.21.15.110:6827/518896551","heartbeat_back_addr":"172.21.15.110:6831/518896551","heartbeat_front_addr":"172.21.15.110:6829/518896551","state":["exists","up"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6809","nonce":3683566991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6811","nonce":3683566991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6815","nonce":3683566991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6813","nonce":3683566991}]},"public_addr":"172.21.15.110:6809/3683566991","cluster_addr":"172.21.15.110:6811/3683566991","heartbeat_back_addr":"172.21.15.110:6815/3683566991","heartbeat_front_addr":"172.21.15.110:6813/3683566991","state":["exists","up"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6801","nonce":2374111370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6803","nonce":2374111370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6807","nonce":2374111370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6805","nonce":2374111370}]},"public_addr":"172.21.15.110:6801/2374111370","cluster_addr":"172.21.15.110:6803/2374111370","heartbeat_back_addr":"172.21.15.110:6807/2374111370","heartbeat_front_addr":"172.21.15.110:6805/2374111370","state":["exists","up"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6827","nonce":1007517023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6829","nonce":1007517023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6833","nonce":1007517023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6831","nonce":1007517023}]},"public_addr":"172.21.15.162:6827/1007517023","cluster_addr":"172.21.15.162:6829/1007517023","heartbeat_back_addr":"172.21.15.162:6833/1007517023","heartbeat_front_addr":"172.21.15.162:6831/1007517023","state":["exists","up"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6803","nonce":946981202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6805","nonce":946981202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6809","nonce":946981202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6807","nonce":946981202}]},"public_addr":"172.21.15.162:6803/946981202","cluster_addr":"172.21.15.162:6805/946981202","heartbeat_back_addr":"172.21.15.162:6809/946981202","heartbeat_front_addr":"172.21.15.162:6807/946981202","state":["exists","up"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6811","nonce":3628034141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6813","nonce":3628034141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6817","nonce":3628034141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6815","nonce":3628034141}]},"public_addr":"172.21.15.162:6811/3628034141","cluster_addr":"172.21.15.162:6813/3628034141","heartbeat_back_addr":"172.21.15.162:6817/3628034141","heartbeat_front_addr":"172.21.15.162:6815/3628034141","state":["exists","up"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6819","nonce":68785612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6821","nonce":68785612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6825","nonce":68785612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6823","nonce":68785612}]},"public_addr":"172.21.15.162:6819/68785612","cluster_addr":"172.21.15.162:6821/68785612","heartbeat_back_addr":"172.21.15.162:6825/68785612","heartbeat_front_addr":"172.21.15.162:6823/68785612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.313498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:10.279187+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.161615+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.904665+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:08.256516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.104354+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:08.722052+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:07.959074+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:43:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 10 librados: watch_flush enter 2024-01-27T03:43:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 10 librados: watch_flush exit 2024-01-27T03:43:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 20 client.4284.objecter shutdown clearing up homeless session... 2024-01-27T03:43:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 10 client.4284.objecter successfully canceled tick 2024-01-27T03:43:31.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb00c06c9a0 msgr2=0x7fb00c06ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb00c06c9a0 0x7fb00c06ee60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0af1a0 tx=0x7fb014009060 comp rx=0 tx=0).stop 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 10 monclient: shutdown 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb02c0b3b30 msgr2=0x7fb02c13ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb02c0b3b30 0x7fb02c13ab00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00c4a0 tx=0x7fb01c00c960 comp rx=0 tx=0).stop 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 shutdown_connections 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb00c06c9a0 0x7fb00c06ee60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb02c0a94b0 0x7fb02c0ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb02c0b4680 0x7fb02c0aa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 --2- 172.21.15.110:0/3430690811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb02c0b3b30 0x7fb02c13ab00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 >> 172.21.15.110:0/3430690811 conn(0x7fb02c066490 msgr2=0x7fb02c0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 shutdown_connections 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 -- 172.21.15.110:0/3430690811 wait complete. 2024-01-27T03:43:31.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.353+0000 7fb0324d4640 1 librados: shutdown 2024-01-27T03:43:31.365 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-27T03:43:31.365 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T03:43:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.463+0000 7f75cc0a2640 10 monclient: get_monmap_and_config 2024-01-27T03:43:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.463+0000 7f75cc0a2640 10 monclient: build_initial_monmap 2024-01-27T03:43:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.463+0000 7f75cc0a2640 10 monclient: monmap: 2024-01-27T03:43:31.464 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:31.464711+0000 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:31.464711+0000 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:31.465 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 1 Processor -- start 2024-01-27T03:43:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 1 -- start start 2024-01-27T03:43:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient: init 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f75c40b4680 0x7f75c4142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient(hunting): picked mon.noname-c con 0x7f75c40b4680 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75c40b4a60 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c4142eb0 0x7f75c40b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient(hunting): picked mon.noname-b con 0x7f75c4142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c40afbe0 con 0x7f75c40b4a60 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c4142240 con 0x7f75c40b4680 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40b0890 con 0x7f75c4142eb0 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:31.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.464+0000 7f75cc0a2640 10 monclient(hunting): authenticate will time out at 1494.818902s 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75c963d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f75c40b4a60 0x7f75c40b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55396/0 (socket says 172.21.15.110:55396) 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75c963d640 1 -- 172.21.15.110:0/4142016701 learned_addr learned my addr 172.21.15.110:0/4142016701 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75ca63f640 1 --2- 172.21.15.110:0/4142016701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c4142eb0 0x7f75c40b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75c9e3e640 1 --2- 172.21.15.110:0/4142016701 >> v2:172.21.15.162:3300/0 conn(0x7f75c40b4680 0x7f75c4142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75ca63f640 10 monclient(hunting): get_auth_request con 0x7f75c4142eb0 auth_method 0 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75ca63f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75ca63f640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75ca63f640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75c9e3e640 10 monclient(hunting): get_auth_request con 0x7f75c40b4680 auth_method 0 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75c9e3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75c9e3e640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.465+0000 7f75c9e3e640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.466+0000 7f75ca63f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.466+0000 7f75ca63f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.466+0000 7f75ca63f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.466+0000 7f75c9e3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.466+0000 7f75c9e3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.466+0000 7f75c9e3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.467+0000 7f75c8e3c640 1 -- 172.21.15.110:0/4142016701 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365018257 0 0) 0x7f75c40afbe0 con 0x7f75c40b4a60 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.467+0000 7f75c8e3c640 10 monclient(hunting): _negotiate 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.467+0000 7f75c8e3c640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.467+0000 7f75c8e3c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.467+0000 7f75c8e3c640 10 monclient(hunting): my global_id is 4287 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.467+0000 7f75c8e3c640 1 -- 172.21.15.110:0/4142016701 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b0003520 con 0x7f75c40b4a60 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.467+0000 7f75ca63f640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.468+0000 7f75ca63f640 1 -- 172.21.15.110:0/4142016701 >> v2:172.21.15.162:3300/0 conn(0x7f75c40b4680 msgr2=0x7f75c4142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.468+0000 7f75ca63f640 1 --2- 172.21.15.110:0/4142016701 >> v2:172.21.15.162:3300/0 conn(0x7f75c40b4680 0x7f75c4142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.468+0000 7f75ca63f640 1 -- 172.21.15.110:0/4142016701 >> v1:172.21.15.110:6789/0 conn(0x7f75c40b4a60 legacy=0x7f75c40b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.468+0000 7f75ca63f640 10 monclient: _finish_hunting 0 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.468+0000 7f75ca63f640 1 monclient: found mon.noname-b 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.468+0000 7f75ca63f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.468+0000 7f75ca63f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.468+0000 7f75ca63f640 1 -- 172.21.15.110:0/4142016701 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c40b0b00 con 0x7f75c4142eb0 2024-01-27T03:43:31.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.469+0000 7f75ca63f640 1 --2- 172.21.15.110:0/4142016701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c4142eb0 0x7f75c40b1b80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f75bc009d90 tx=0x7f75bc02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f404c456f9dfe4c server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 1 -- 172.21.15.110:0/4142016701 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc02cbc0 con 0x7f75c4142eb0 2024-01-27T03:43:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 10 monclient: dump: 2024-01-27T03:43:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:31.474 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 10 monclient: _finish_auth 0 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 10 monclient: _check_auth_tickets 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 1 -- 172.21.15.110:0/4142016701 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc02cd20 con 0x7f75c4142eb0 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 1 -- 172.21.15.110:0/4142016701 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc034660 con 0x7f75c4142eb0 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:31.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75c8e3c640 10 monclient: dump: 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75cc0a2640 5 monclient: authenticate success, global_id 4256 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75cc0a2640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:31.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.473+0000 7f75cc0a2640 4 set_mon_vals no callback set 2024-01-27T03:43:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 10 monclient: shutdown 2024-01-27T03:43:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 1 -- 172.21.15.110:0/4142016701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c4142eb0 msgr2=0x7f75c40b1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 1 --2- 172.21.15.110:0/4142016701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c4142eb0 0x7f75c40b1b80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f75bc009d90 tx=0x7f75bc02be10 comp rx=0 tx=0).stop 2024-01-27T03:43:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 1 -- 172.21.15.110:0/4142016701 shutdown_connections 2024-01-27T03:43:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 1 --2- 172.21.15.110:0/4142016701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c4142eb0 0x7f75c40b1b80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 1 --2- 172.21.15.110:0/4142016701 >> v2:172.21.15.162:3300/0 conn(0x7f75c40b4680 0x7f75c4142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 1 -- 172.21.15.110:0/4142016701 >> 172.21.15.110:0/4142016701 conn(0x7f75c4066490 msgr2=0x7f75c40668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:31.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 1 -- 172.21.15.110:0/4142016701 shutdown_connections 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.474+0000 7f75cc0a2640 1 -- 172.21.15.110:0/4142016701 wait complete. 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.475+0000 7f75cc0a2640 10 monclient: build_initial_monmap 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.475+0000 7f75cc0a2640 10 monclient: monmap: 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.475+0000 7f75cc0a2640 1 librados: starting msgr at 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.475+0000 7f75cc0a2640 1 librados: starting objecter 2024-01-27T03:43:31.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.475+0000 7f75cc0a2640 1 Processor -- start 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.475+0000 7f75cc0a2640 1 -- start start 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.475+0000 7f75cc0a2640 1 librados: setting wanted keys 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.475+0000 7f75cc0a2640 1 librados: calling monclient init 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.475+0000 7f75cc0a2640 10 monclient: init 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40b4680 0x7f75c413d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient(hunting): picked mon.noname-c con 0x7f75c40b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4142eb0 0x7f75c413dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient(hunting): picked mon.noname-a con 0x7f75c4142eb0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c413a780 0x7f75c413bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient(hunting): picked mon.noname-b con 0x7f75c413a780 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:31.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c413db40 con 0x7f75c4142eb0 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c413e510 con 0x7f75c413a780 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c413ef70 con 0x7f75c40b4680 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75cc0a2640 10 monclient(hunting): authenticate will time out at 1494.830057s 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c963d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4142eb0 0x7f75c413dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c9e3e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40b4680 0x7f75c413d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75ca63f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c413a780 0x7f75c413bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c963d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4142eb0 0x7f75c413dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:49948/0 (socket says 172.21.15.110:49948) 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c963d640 1 -- 172.21.15.110:0/3354871484 learned_addr learned my addr 172.21.15.110:0/3354871484 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c963d640 10 monclient(hunting): get_auth_request con 0x7f75c4142eb0 auth_method 0 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c963d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c963d640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:31.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c963d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75ca63f640 10 monclient(hunting): get_auth_request con 0x7f75c413a780 auth_method 0 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75ca63f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75ca63f640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75ca63f640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c9e3e640 10 monclient(hunting): get_auth_request con 0x7f75c40b4680 auth_method 0 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c9e3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c9e3e640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.476+0000 7f75c9e3e640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.477+0000 7f75ca63f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.477+0000 7f75ca63f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.477+0000 7f75ca63f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.477+0000 7f75c963d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.477+0000 7f75c963d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.477+0000 7f75c963d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:31.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.477+0000 7f75c9e3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.477+0000 7f75c9e3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.477+0000 7f75c9e3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 10 monclient(hunting): handle_auth_done global_id 4291 payload 824 2024-01-27T03:43:31.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 1 -- 172.21.15.110:0/3354871484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40b4680 msgr2=0x7f75c413d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40b4680 0x7f75c413d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 1 -- 172.21.15.110:0/3354871484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4142eb0 msgr2=0x7f75c413dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4142eb0 0x7f75c413dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 10 monclient: _finish_hunting 0 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 1 monclient: found mon.noname-b 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.479+0000 7f75ca63f640 1 -- 172.21.15.110:0/3354871484 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc009a70 con 0x7f75c413a780 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.482+0000 7f75ca63f640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c413a780 0x7f75c413bcc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f75bc033000 tx=0x7f75bc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.482+0000 7f75ba7fc640 10 client.?.objecter ms_handle_connect 0x7f75c413a780 2024-01-27T03:43:31.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.482+0000 7f75ba7fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:31.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 1 -- 172.21.15.110:0/3354871484 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc040650 con 0x7f75c413a780 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 10 monclient: dump: 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:31.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 1 -- 172.21.15.110:0/3354871484 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc02ce90 con 0x7f75c413a780 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 1 -- 172.21.15.110:0/3354871484 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc0409f0 con 0x7f75c413a780 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75ba7fc640 10 monclient: dump: 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 5 monclient: authenticate success, global_id 4291 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 10 monclient: _renew_subs 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c413f250 con 0x7f75c413a780 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.485+0000 7f75bbfff640 4 set_mon_vals no callback set 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 10 monclient: _renew_subs 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c413c200 con 0x7f75c413a780 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 1 librados: init done 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.486+0000 7f75cc0a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.487+0000 7f75cc0a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.487+0000 7f75cc0a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.487+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c4066da0 con 0x7f75c413a780 2024-01-27T03:43:31.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.505+0000 7f75ba7fc640 1 -- 172.21.15.110:0/3354871484 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f75bc040b50 con 0x7f75c413a780 2024-01-27T03:43:31.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.505+0000 7f75ba7fc640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f75a806c9a0 0x7f75a806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:31.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.505+0000 7f75c9e3e640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f75a806c9a0 0x7f75a806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:31.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.505+0000 7f75c9e3e640 10 monclient: get_auth_request con 0x7f75a806c9a0 auth_method 0 2024-01-27T03:43:31.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.506+0000 7f75c9e3e640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f75a806c9a0 0x7f75a806ee60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009b40 tx=0x7f75c0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:31.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.506+0000 7f75ba7fc640 10 client.4291.objecter ms_handle_connect 0x7f75a806c9a0 2024-01-27T03:43:31.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.507+0000 7f75ba7fc640 1 -- 172.21.15.110:0/3354871484 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f75bc07b2f0 con 0x7f75c413a780 2024-01-27T03:43:31.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.507+0000 7f75ba7fc640 10 client.4291.objecter ms_dispatch 0x7f75c4000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:31.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.507+0000 7f75ba7fc640 3 client.4291.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:31.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.507+0000 7f75ba7fc640 3 client.4291.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:31.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.507+0000 7f75ba7fc640 20 client.4291.objecter dump_active .. 0 homeless 2024-01-27T03:43:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.747+0000 7f75ba7fc640 1 -- 172.21.15.110:0/3354871484 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f75bc077590 con 0x7f75c413a780 2024-01-27T03:43:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.747+0000 7f75ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.747+0000 7f75ba7fc640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:31.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.983+0000 7f75cc0a2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.983+0000 7f75cc0a2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.983+0000 7f75cc0a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:31.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:31.983+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75c404ea50 con 0x7f75c413a780 2024-01-27T03:43:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.017+0000 7f75ba7fc640 1 -- 172.21.15.110:0/3354871484 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14675 (secure 0 0 0) 0x7f75bc044090 con 0x7f75c413a780 2024-01-27T03:43:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.017+0000 7f75ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T03:43:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.017+0000 7f75ba7fc640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:32.019 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:43:32.019 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":26,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T03:43:30.682666+0000","last_up_change":"2024-01-27T03:43:21.487334+0000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-27T03:43:13.074487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-27T03:43:27.987244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6817","nonce":1545415171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6819","nonce":1545415171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6824","nonce":1545415171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6821","nonce":1545415171}]},"public_addr":"172.21.15.110:6817/1545415171","cluster_addr":"172.21.15.110:6819/1545415171","heartbeat_back_addr":"172.21.15.110:6824/1545415171","heartbeat_front_addr":"172.21.15.110:6821/1545415171","state":["exists","up"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6823","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6825","nonce":518896551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6827","nonce":518896551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6831","nonce":518896551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6829","nonce":518896551}]},"public_addr":"172.21.15.110:6825/518896551","cluster_addr":"172.21.15.110:6827/518896551","heartbeat_back_addr":"172.21.15.110:6831/518896551","heartbeat_front_addr":"172.21.15.110:6829/518896551","state":["exists","up"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6809","nonce":3683566991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6811","nonce":3683566991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6815","nonce":3683566991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6813","nonce":3683566991}]},"public_addr":"172.21.15.110:6809/3683566991","cluster_addr":"172.21.15.110:6811/3683566991","heartbeat_back_addr":"172.21.15.110:6815/3683566991","heartbeat_front_addr":"172.21.15.110:6813/3683566991","state":["exists","up"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6801","nonce":2374111370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6803","nonce":2374111370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6807","nonce":2374111370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6805","nonce":2374111370}]},"public_addr":"172.21.15.110:6801/2374111370","cluster_addr":"172.21.15.110:6803/2374111370","heartbeat_back_addr":"172.21.15.110:6807/2374111370","heartbeat_front_addr":"172.21.15.110:6805/2374111370","state":["exists","up"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6827","nonce":1007517023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6829","nonce":1007517023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6833","nonce":1007517023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6831","nonce":1007517023}]},"public_addr":"172.21.15.162:6827/1007517023","cluster_addr":"172.21.15.162:6829/1007517023","heartbeat_back_addr":"172.21.15.162:6833/1007517023","heartbeat_front_addr":"172.21.15.162:6831/1007517023","state":["exists","up"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6803","nonce":946981202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6805","nonce":946981202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6809","nonce":946981202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6807","nonce":946981202}]},"public_addr":"172.21.15.162:6803/946981202","cluster_addr":"172.21.15.162:6805/946981202","heartbeat_back_addr":"172.21.15.162:6809/946981202","heartbeat_front_addr":"172.21.15.162:6807/946981202","state":["exists","up"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6811","nonce":3628034141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6813","nonce":3628034141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6817","nonce":3628034141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6815","nonce":3628034141}]},"public_addr":"172.21.15.162:6811/3628034141","cluster_addr":"172.21.15.162:6813/3628034141","heartbeat_back_addr":"172.21.15.162:6817/3628034141","heartbeat_front_addr":"172.21.15.162:6815/3628034141","state":["exists","up"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6819","nonce":68785612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6821","nonce":68785612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6825","nonce":68785612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6823","nonce":68785612}]},"public_addr":"172.21.15.162:6819/68785612","cluster_addr":"172.21.15.162:6821/68785612","heartbeat_back_addr":"172.21.15.162:6825/68785612","heartbeat_front_addr":"172.21.15.162:6823/68785612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.313498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:10.279187+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.161615+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.904665+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:08.256516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.104354+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:08.722052+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:07.959074+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T03:43:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 10 librados: watch_flush enter 2024-01-27T03:43:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 10 librados: watch_flush exit 2024-01-27T03:43:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 20 client.4291.objecter shutdown clearing up homeless session... 2024-01-27T03:43:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 10 client.4291.objecter successfully canceled tick 2024-01-27T03:43:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f75a806c9a0 msgr2=0x7f75a806ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f75a806c9a0 0x7f75a806ee60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009b40 tx=0x7f75c0009190 comp rx=0 tx=0).stop 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 10 monclient: shutdown 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c413a780 msgr2=0x7f75c413bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c413a780 0x7f75c413bcc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f75bc033000 tx=0x7f75bc004a40 comp rx=0 tx=0).stop 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 shutdown_connections 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f75a806c9a0 0x7f75a806ee60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c413a780 0x7f75c413bcc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4142eb0 0x7f75c413dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 --2- 172.21.15.110:0/3354871484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40b4680 0x7f75c413d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 >> 172.21.15.110:0/3354871484 conn(0x7f75c4066490 msgr2=0x7f75c40b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.019+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 shutdown_connections 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.020+0000 7f75cc0a2640 1 -- 172.21.15.110:0/3354871484 wait complete. 2024-01-27T03:43:32.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.020+0000 7f75cc0a2640 1 librados: shutdown 2024-01-27T03:43:32.032 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-27T03:43:32.032 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-27T03:43:32.033 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-27T03:43:32.033 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-27T03:43:32.034 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-27T03:43:32.034 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-27T03:43:32.035 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-27T03:43:32.035 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-27T03:43:32.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7ff9fed9e640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7ff9fed9e640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7ff9fed9e640 10 monclient: monmap: 2024-01-27T03:43:32.203 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.203 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.203 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.203337+0000 2024-01-27T03:43:32.203 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.203337+0000 2024-01-27T03:43:32.203 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.203 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.204 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.204 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.204 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.204 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.204 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.204 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7f6c0bc3f640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7f6c0bc3f640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7f6c0bc3f640 10 monclient: monmap: 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.203696+0000 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.203696+0000 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.205 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7ff40c11d640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7ff40c11d640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7ff40c11d640 10 monclient: monmap: 2024-01-27T03:43:32.206 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.206 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.206 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.203657+0000 2024-01-27T03:43:32.206 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.203657+0000 2024-01-27T03:43:32.206 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.206 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.207 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.207 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.207 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.207 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.207 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.207 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7f119b29a640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7f119b29a640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.202+0000 7f119b29a640 10 monclient: monmap: 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.203949+0000 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.203949+0000 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.209 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7ff9fed9e640 1 Processor -- start 2024-01-27T03:43:32.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7ff9fed9e640 1 -- start start 2024-01-27T03:43:32.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7ff9fed9e640 10 monclient: init 2024-01-27T03:43:32.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7f119b29a640 1 Processor -- start 2024-01-27T03:43:32.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7f78084f7640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7f78084f7640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7f78084f7640 10 monclient: monmap: 2024-01-27T03:43:32.210 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.210 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.210 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.205043+0000 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.205043+0000 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7f119b29a640 1 -- start start 2024-01-27T03:43:32.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7f119b29a640 10 monclient: init 2024-01-27T03:43:32.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f567ffff640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9f80b4680 0x7ff9f8142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f80b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9f80b4a60 0x7ff9f80b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f80b4a60 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff9f8142eb0 0x7ff9f80b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient(hunting): picked mon.noname-c con 0x7ff9f8142eb0 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f8142240 con 0x7ff9f80b4a60 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f80afbe0 con 0x7ff9f8142eb0 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80b0890 con 0x7ff9f80b4680 2024-01-27T03:43:32.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff9fed9e640 10 monclient(hunting): authenticate will time out at 1495.558416s 2024-01-27T03:43:32.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f567ffff640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.203+0000 7ff40c11d640 1 Processor -- start 2024-01-27T03:43:32.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11940ae150 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1194072a50 0x7f11940aed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 10 monclient(hunting): picked mon.noname-c con 0x7f1194072a50 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11940acef0 0x7f11940ada30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 10 monclient(hunting): picked mon.noname-b con 0x7f11940acef0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11940ac7f0 con 0x7f11940ae150 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11940ad510 con 0x7f1194072a50 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11940aab30 con 0x7f11940acef0 2024-01-27T03:43:32.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f567ffff640 10 monclient: monmap: 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.205702+0000 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.205702+0000 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.217 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.218 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7f119b29a640 10 monclient(hunting): authenticate will time out at 1495.558773s 2024-01-27T03:43:32.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7f6c0bc3f640 1 Processor -- start 2024-01-27T03:43:32.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7f6c0bc3f640 1 -- start start 2024-01-27T03:43:32.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7f6c0bc3f640 10 monclient: init 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.204+0000 7ff40c11d640 1 -- start start 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient: init 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9fd33b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff9f8142eb0 0x7ff9f80b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9f7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9f80b4a60 0x7ff9f80b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9f7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9f80b4a60 0x7ff9f80b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:49962/0 (socket says 172.21.15.110:49962) 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9fcb3a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9f80b4680 0x7ff9f8142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9f7fff640 1 -- 172.21.15.110:0/3514347205 learned_addr learned my addr 172.21.15.110:0/3514347205 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9f7fff640 10 monclient(hunting): get_auth_request con 0x7ff9f80b4a60 auth_method 0 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-01-27T03:43:32.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff404072100 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:32.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff4040b0ab0 0x7ff404149950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4040b0ab0 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff40414ecd0 0x7ff4041510c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient(hunting): picked mon.noname-b con 0x7ff40414ecd0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40406fc60 con 0x7ff404072100 2024-01-27T03:43:32.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff404144070 con 0x7ff4040b0ab0 2024-01-27T03:43:32.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40406ea70 con 0x7ff40414ecd0 2024-01-27T03:43:32.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7ff40c11d640 10 monclient(hunting): authenticate will time out at 1495.559935s 2024-01-27T03:43:32.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7f1199a97640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1194072a50 0x7f11940aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f119a298640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f11940ae150 0x7f11940a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:49102/0 (socket says 172.21.15.110:49102) 2024-01-27T03:43:32.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f119a298640 1 -- 172.21.15.110:0/3310602383 learned_addr learned my addr 172.21.15.110:0/3310602383 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7f78084f7640 1 Processor -- start 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7f78084f7640 1 -- start start 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.205+0000 7f78084f7640 10 monclient: init 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f7800143930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 10 monclient(hunting): picked mon.noname-b con 0x7f78000b4930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78000b41f0 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7800143e70 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7800144300 con 0x7f78000b41f0 2024-01-27T03:43:32.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78000af7e0 con 0x7f7800143e70 2024-01-27T03:43:32.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78000b0b00 con 0x7f78000b4930 2024-01-27T03:43:32.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f78084f7640 10 monclient(hunting): authenticate will time out at 1495.560384s 2024-01-27T03:43:32.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f567ffff640 1 Processor -- start 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7fd23733e640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7fd23733e640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7fd23733e640 10 monclient: monmap: 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.207659+0000 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.207659+0000 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.224 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.225 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f1199a97640 10 monclient(hunting): get_auth_request con 0x7f1194072a50 auth_method 0 2024-01-27T03:43:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f1199a97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f1199a97640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f1199a97640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-01-27T03:43:32.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6bfc0c32d0 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:32.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bfc0c9aa0 0x7f6bfc0c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 10 monclient(hunting): picked mon.noname-a con 0x7f6bfc0c9aa0 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:32.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6bfc0c5a20 0x7f6bfc0d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 10 monclient(hunting): picked mon.noname-c con 0x7f6bfc0c5a20 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bfc0c9580 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bfc031e60 con 0x7f6bfc0c9aa0 2024-01-27T03:43:32.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bfc0c36b0 con 0x7f6bfc0c5a20 2024-01-27T03:43:32.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f6c0bc3f640 10 monclient(hunting): authenticate will time out at 1495.560690s 2024-01-27T03:43:32.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f119aa99640 1 --2- 172.21.15.110:0/3310602383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11940acef0 0x7f11940ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f567ffff640 1 -- start start 2024-01-27T03:43:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f567ffff640 10 monclient: init 2024-01-27T03:43:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9fcb3a640 10 monclient(hunting): get_auth_request con 0x7ff9f80b4680 auth_method 0 2024-01-27T03:43:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9fcb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9fcb3a640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7ff9fcb3a640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.206+0000 7f119aa99640 10 monclient(hunting): get_auth_request con 0x7f11940acef0 auth_method 0 2024-01-27T03:43:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f119aa99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f119aa99640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7ff9f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7ff9f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f119aa99640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7ff9f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f1199a97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f1199a97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f1199a97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7fd23733e640 1 Processor -- start 2024-01-27T03:43:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7fd23733e640 1 -- start start 2024-01-27T03:43:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7fd23733e640 10 monclient: init 2024-01-27T03:43:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f119aa99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f119aa99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f119aa99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-01-27T03:43:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5678039020 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5678034d00 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:32.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5678036380 0x7f56780d7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7ff4096b8640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff4040b0ab0 0x7ff404149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f5678036380 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567800e0f0 con 0x7f5678039020 2024-01-27T03:43:32.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56780373a0 con 0x7f5678034d00 2024-01-27T03:43:32.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5678039400 con 0x7f5678036380 2024-01-27T03:43:32.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7f567ffff640 10 monclient(hunting): authenticate will time out at 1495.561911s 2024-01-27T03:43:32.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.207+0000 7ff4096b8640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff4040b0ab0 0x7ff404149950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40564/0 (socket says 172.21.15.110:40564) 2024-01-27T03:43:32.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7ff4096b8640 1 -- 172.21.15.110:0/578327427 learned_addr learned my addr 172.21.15.110:0/578327427 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7ff4096b8640 10 monclient(hunting): get_auth_request con 0x7ff4040b0ab0 auth_method 0 2024-01-27T03:43:32.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7ff4096b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7ff4096b8640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7ff4096b8640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-01-27T03:43:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2300b4680 0x7fd230142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2300b4680 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300b4a60 0x7fd2300b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient(hunting): picked mon.noname-b con 0x7fd2300b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd230142eb0 0x7fd2300b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient(hunting): picked mon.noname-c con 0x7fd230142eb0 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd230142240 con 0x7fd2300b4680 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2300afbe0 con 0x7fd230142eb0 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300b0890 con 0x7fd2300b4a60 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7fd23733e640 10 monclient(hunting): authenticate will time out at 1495.562473s 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7ff9fcb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7ff9fcb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7ff9fcb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7f0563fff640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7f0563fff640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.208+0000 7f0563fff640 10 monclient: monmap: 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.210089+0000 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.210089+0000 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.239 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f1199a97640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2024-01-27T03:43:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f1199a97640 1 -- 172.21.15.110:0/3310602383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11940acef0 msgr2=0x7f11940ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f1199a97640 1 --2- 172.21.15.110:0/3310602383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11940acef0 0x7f11940ada30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f1199a97640 1 -- 172.21.15.110:0/3310602383 >> v1:172.21.15.162:6789/0 conn(0x7f11940ae150 legacy=0x7f11940a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f1199a97640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f1199a97640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f1199a97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f1199a97640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f1199a97640 1 -- 172.21.15.110:0/3310602383 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940aaca0 con 0x7f1194072a50 2024-01-27T03:43:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-01-27T03:43:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 1 -- 172.21.15.110:0/3514347205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9f80b4680 msgr2=0x7ff9f8142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 1 --2- 172.21.15.110:0/3514347205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9f80b4680 0x7ff9f8142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 1 -- 172.21.15.110:0/3514347205 >> v2:172.21.15.162:3300/0 conn(0x7ff9f8142eb0 msgr2=0x7ff9f80b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 1 --2- 172.21.15.110:0/3514347205 >> v2:172.21.15.162:3300/0 conn(0x7ff9f8142eb0 0x7ff9f80b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 1 monclient: found mon.noname-a 2024-01-27T03:43:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7ff9f7fff640 1 -- 172.21.15.110:0/3514347205 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f80b0b00 con 0x7ff9f80b4a60 2024-01-27T03:43:32.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f0563fff640 1 Processor -- start 2024-01-27T03:43:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f0563fff640 1 -- start start 2024-01-27T03:43:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.209+0000 7f0563fff640 10 monclient: init 2024-01-27T03:43:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-01-27T03:43:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f055c006fe0 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7ff4096b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7ff4096b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7ff4096b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f055c0073c0 0x7f055c0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 10 monclient(hunting): picked mon.noname-c con 0x7f055c0073c0 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f055c031740 0x7f055c0d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 10 monclient(hunting): picked mon.noname-a con 0x7f055c031740 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055c036fa0 con 0x7f055c006fe0 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055c037b40 con 0x7f055c031740 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055c034ea0 con 0x7f055c0073c0 2024-01-27T03:43:32.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.210+0000 7f0563fff640 10 monclient(hunting): authenticate will time out at 1495.564805s 2024-01-27T03:43:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.211+0000 7f6c091da640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bfc0c9aa0 0x7f6bfc0c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.211+0000 7f6c091da640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bfc0c9aa0 0x7f6bfc0c8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:49968/0 (socket says 172.21.15.110:49968) 2024-01-27T03:43:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.211+0000 7f6c091da640 1 -- 172.21.15.110:0/2396355139 learned_addr learned my addr 172.21.15.110:0/2396355139 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.211+0000 7f6c091da640 10 monclient(hunting): get_auth_request con 0x7f6bfc0c9aa0 auth_method 0 2024-01-27T03:43:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.211+0000 7f6c091da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.211+0000 7f6c091da640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.211+0000 7f6c091da640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.211+0000 7fd2350da640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2300b4680 0x7fd230142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7f6c091da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7f6c091da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7f6c091da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7fd2350da640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2300b4680 0x7fd230142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:49970/0 (socket says 172.21.15.110:49970) 2024-01-27T03:43:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7fd2350da640 1 -- 172.21.15.110:0/513500446 learned_addr learned my addr 172.21.15.110:0/513500446 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7fd2350da640 10 monclient(hunting): get_auth_request con 0x7fd2300b4680 auth_method 0 2024-01-27T03:43:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7fd2350da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7fd2350da640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7fd2350da640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7ff4096b8640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-01-27T03:43:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7ff4096b8640 1 -- 172.21.15.110:0/578327427 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff40414ecd0 msgr2=0x7ff4041510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7ff4096b8640 1 --2- 172.21.15.110:0/578327427 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff40414ecd0 0x7ff4041510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7ff4096b8640 1 -- 172.21.15.110:0/578327427 >> v1:172.21.15.162:6789/0 conn(0x7ff404072100 legacy=0x7ff40406a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7ff4096b8640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7ff4096b8640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7ff4096b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7ff4096b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.212+0000 7ff4096b8640 1 -- 172.21.15.110:0/578327427 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40406ed50 con 0x7ff4040b0ab0 2024-01-27T03:43:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.213+0000 7fd2350da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.213+0000 7fd2350da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.213+0000 7fd2350da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.214+0000 7f1199a97640 1 --2- 172.21.15.110:0/3310602383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1194072a50 0x7f11940aed60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f11900097a0 tx=0x7f119002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5f8a80bb7e233da server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.214+0000 7f7806293640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f7800143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.214+0000 7f6c091da640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-01-27T03:43:32.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.214+0000 7f6c091da640 1 -- 172.21.15.110:0/2396355139 >> v2:172.21.15.162:3300/0 conn(0x7f6bfc0c5a20 msgr2=0x7f6bfc0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.215+0000 7f6c091da640 1 --2- 172.21.15.110:0/2396355139 >> v2:172.21.15.162:3300/0 conn(0x7f6bfc0c5a20 0x7f6bfc0d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.215+0000 7f6c091da640 1 -- 172.21.15.110:0/2396355139 >> v1:172.21.15.110:6789/0 conn(0x7f6bfc0c32d0 legacy=0x7f6bfc006d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.215+0000 7f6c091da640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.215+0000 7f6c091da640 1 monclient: found mon.noname-a 2024-01-27T03:43:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.215+0000 7f6c091da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.215+0000 7f6c091da640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.215+0000 7f6c091da640 1 -- 172.21.15.110:0/2396355139 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc02ea20 con 0x7f6bfc0c9aa0 2024-01-27T03:43:32.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.216+0000 7ff4096b8640 1 --2- 172.21.15.110:0/578327427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4040b0ab0 0x7ff404149950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009870 tx=0x7ff3f402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb8b0380d8fe9f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.216+0000 7fd2348d9640 1 --2- 172.21.15.110:0/513500446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300b4a60 0x7fd2300b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.216+0000 7fd2348d9640 10 monclient(hunting): get_auth_request con 0x7fd2300b4a60 auth_method 0 2024-01-27T03:43:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.216+0000 7fd2348d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.216+0000 7fd2348d9640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.216+0000 7fd2348d9640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-01-27T03:43:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 1 -- 172.21.15.110:0/513500446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300b4a60 msgr2=0x7fd2300b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 1 --2- 172.21.15.110:0/513500446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300b4a60 0x7fd2300b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 1 -- 172.21.15.110:0/513500446 >> v2:172.21.15.162:3300/0 conn(0x7fd230142eb0 msgr2=0x7fd2300b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 1 --2- 172.21.15.110:0/513500446 >> v2:172.21.15.162:3300/0 conn(0x7fd230142eb0 0x7fd2300b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 1 monclient: found mon.noname-a 2024-01-27T03:43:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7fd2350da640 1 -- 172.21.15.110:0/513500446 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2300b0b00 con 0x7fd2300b4680 2024-01-27T03:43:32.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7f7806293640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f7800143930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38330/0 (socket says 172.21.15.110:38330) 2024-01-27T03:43:32.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7f7806293640 1 -- 172.21.15.110:0/1871336350 learned_addr learned my addr 172.21.15.110:0/1871336350 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7f7806293640 10 monclient(hunting): get_auth_request con 0x7f78000b4930 auth_method 0 2024-01-27T03:43:32.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7f7806293640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7f7806293640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.217+0000 7f7806293640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.219+0000 7f7806293640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.219+0000 7f7806293640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.219+0000 7f7806293640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.219+0000 7f567e7fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f5678034d00 0x7f5678035c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:49136/0 (socket says 172.21.15.110:49136) 2024-01-27T03:43:32.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.219+0000 7f567e7fc640 1 -- 172.21.15.110:0/2408442023 learned_addr learned my addr 172.21.15.110:0/2408442023 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:32.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.220+0000 7f6c091da640 1 --2- 172.21.15.110:0/2396355139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bfc0c9aa0 0x7f6bfc0c8f60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00c9f0 tx=0x7f6c0002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bfd640fe1b0107 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.221+0000 7f7806293640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-01-27T03:43:32.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.221+0000 7f7806293640 1 -- 172.21.15.110:0/1871336350 >> v1:172.21.15.162:6789/0 conn(0x7f7800143e70 legacy=0x7f7800150f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.221+0000 7f7806293640 1 -- 172.21.15.110:0/1871336350 >> v1:172.21.15.110:6789/0 conn(0x7f78000b41f0 legacy=0x7f78000b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.221+0000 7f7806293640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.221+0000 7f7806293640 1 monclient: found mon.noname-b 2024-01-27T03:43:32.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.221+0000 7f7806293640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.221+0000 7f7806293640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.221+0000 7f7806293640 1 -- 172.21.15.110:0/1871336350 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78000b0de0 con 0x7f78000b4930 2024-01-27T03:43:32.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.223+0000 7f7806293640 1 --2- 172.21.15.110:0/1871336350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f7800143930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0098a0 tx=0x7f77fc02b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f48a78940bbe63c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.223+0000 7fd2350da640 1 --2- 172.21.15.110:0/513500446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2300b4680 0x7fd230142970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2300b4560 tx=0x7fd22c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9957eaac5082844 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.224+0000 7f05627fc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f055c0073c0 0x7f055c0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.224+0000 7f05627fc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f055c0073c0 0x7f055c0c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40612/0 (socket says 172.21.15.110:40612) 2024-01-27T03:43:32.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.224+0000 7f05627fc640 1 -- 172.21.15.110:0/3363262173 learned_addr learned my addr 172.21.15.110:0/3363262173 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.224+0000 7f05627fc640 10 monclient(hunting): get_auth_request con 0x7f055c0073c0 auth_method 0 2024-01-27T03:43:32.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.224+0000 7f05627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.224+0000 7f05627fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.224+0000 7f05627fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.225+0000 7ff9f7fff640 1 --2- 172.21.15.110:0/3514347205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9f80b4a60 0x7ff9f80b1470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff9f80b4560 tx=0x7ff9ec02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde16c485d95ca21 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 1 -- 172.21.15.110:0/1871336350 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc031520 con 0x7f78000b4930 2024-01-27T03:43:32.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:32.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 10 monclient: dump: 2024-01-27T03:43:32.261 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 1 -- 172.21.15.110:0/1871336350 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc031680 con 0x7f78000b4930 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 1 -- 172.21.15.110:0/1871336350 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc031a10 con 0x7f78000b4930 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f7805291640 10 monclient: dump: 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.263 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f78084f7640 5 monclient: authenticate success, global_id 4271 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f78084f7640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.227+0000 7f78084f7640 4 set_mon_vals no callback set 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 10 monclient: shutdown 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 1 -- 172.21.15.110:0/1871336350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 msgr2=0x7f7800143930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 1 --2- 172.21.15.110:0/1871336350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f7800143930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0098a0 tx=0x7f77fc02b7c0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 1 -- 172.21.15.110:0/1871336350 shutdown_connections 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 1 --2- 172.21.15.110:0/1871336350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f7800143930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 1 -- 172.21.15.110:0/1871336350 >> 172.21.15.110:0/1871336350 conn(0x7f7800066490 msgr2=0x7f78000668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 1 -- 172.21.15.110:0/1871336350 shutdown_connections 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 1 -- 172.21.15.110:0/1871336350 wait complete. 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.228+0000 7f78084f7640 10 monclient: monmap: 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.229+0000 7f78084f7640 1 librados: starting msgr at 2024-01-27T03:43:32.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.229+0000 7f78084f7640 1 librados: starting objecter 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.229+0000 7f78084f7640 1 Processor -- start 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.229+0000 7f78084f7640 1 -- start start 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.229+0000 7f78084f7640 1 librados: setting wanted keys 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.229+0000 7f78084f7640 1 librados: calling monclient init 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.229+0000 7f78084f7640 10 monclient: init 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78000b41f0 0x7f78001cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient(hunting): picked mon.noname-b con 0x7f78000b41f0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f780013d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient(hunting): picked mon.noname-c con 0x7f78000b4930 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7800143e70 0x7f780013de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient(hunting): picked mon.noname-a con 0x7f7800143e70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78001d25e0 con 0x7f7800143e70 2024-01-27T03:43:32.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78001d1820 con 0x7f78000b41f0 2024-01-27T03:43:32.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78001d1b00 con 0x7f78000b4930 2024-01-27T03:43:32.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f78084f7640 10 monclient(hunting): authenticate will time out at 1495.584030s 2024-01-27T03:43:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 1 -- 172.21.15.110:0/578327427 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f40317c0 con 0x7ff4040b0ab0 2024-01-27T03:43:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 10 monclient: dump: 2024-01-27T03:43:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.268 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff40c11d640 5 monclient: authenticate success, global_id 4297 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff40c11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 1 -- 172.21.15.110:0/578327427 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f4031920 con 0x7ff4040b0ab0 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff40c11d640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff40c11d640 4 set_mon_vals no callback set 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 1 -- 172.21.15.110:0/578327427 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4031cb0 con 0x7ff4040b0ab0 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff408eb7640 10 monclient: dump: 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.270 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.271 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.271 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.271 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f567f7fe640 1 --2- 172.21.15.110:0/2408442023 >> v2:172.21.15.162:3300/0 conn(0x7f5678036380 0x7f56780d7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff40c11d640 10 monclient: shutdown 2024-01-27T03:43:32.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff40c11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff40c11d640 1 -- 172.21.15.110:0/578327427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4040b0ab0 msgr2=0x7ff404149950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7ff40c11d640 1 --2- 172.21.15.110:0/578327427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4040b0ab0 0x7ff404149950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009870 tx=0x7ff3f402b860 comp rx=0 tx=0).stop 2024-01-27T03:43:32.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.230+0000 7f7805a92640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f780013d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f7805a92640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f780013d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38354/0 (socket says 172.21.15.110:38354) 2024-01-27T03:43:32.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f7805a92640 1 -- 172.21.15.110:0/259790215 learned_addr learned my addr 172.21.15.110:0/259790215 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7ff40c11d640 1 -- 172.21.15.110:0/578327427 shutdown_connections 2024-01-27T03:43:32.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7ff40c11d640 1 --2- 172.21.15.110:0/578327427 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff40414ecd0 0x7ff4041510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f7805a92640 10 monclient(hunting): get_auth_request con 0x7f78000b4930 auth_method 0 2024-01-27T03:43:32.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f7805a92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f7805a92640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f7805a92640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7ff40c11d640 1 --2- 172.21.15.110:0/578327427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4040b0ab0 0x7ff404149950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7ff40c11d640 1 -- 172.21.15.110:0/578327427 >> 172.21.15.110:0/578327427 conn(0x7ff4040b1400 msgr2=0x7ff4040b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7ff40c11d640 1 -- 172.21.15.110:0/578327427 shutdown_connections 2024-01-27T03:43:32.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7ff40c11d640 1 -- 172.21.15.110:0/578327427 wait complete. 2024-01-27T03:43:32.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f05627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f05627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f05627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f7805a92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f7805a92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7f7805a92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7ff40c11d640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.231+0000 7ff40c11d640 10 monclient: monmap: 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 librados: starting msgr at 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 librados: starting objecter 2024-01-27T03:43:32.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 Processor -- start 2024-01-27T03:43:32.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 -- start start 2024-01-27T03:43:32.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 librados: setting wanted keys 2024-01-27T03:43:32.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 librados: calling monclient init 2024-01-27T03:43:32.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient: init 2024-01-27T03:43:32.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T03:43:32.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404072100 0x7ff404145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient(hunting): picked mon.noname-a con 0x7ff404072100 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7f05627fc640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2024-01-27T03:43:32.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7f05627fc640 1 -- 172.21.15.110:0/3363262173 >> v2:172.21.15.110:3300/0 conn(0x7f055c031740 msgr2=0x7f055c0d0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7f05627fc640 1 --2- 172.21.15.110:0/3363262173 >> v2:172.21.15.110:3300/0 conn(0x7f055c031740 0x7f055c0d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7f05627fc640 1 -- 172.21.15.110:0/3363262173 >> v1:172.21.15.110:6789/0 conn(0x7f055c006fe0 legacy=0x7f055c037430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7f05627fc640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7f05627fc640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7f05627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7f05627fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f05627fc640 1 -- 172.21.15.110:0/3363262173 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055c0c3680 con 0x7f055c0073c0 2024-01-27T03:43:32.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 1 -- 172.21.15.110:0/2396355139 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c000317e0 con 0x7f6bfc0c9aa0 2024-01-27T03:43:32.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:32.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 10 monclient: dump: 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 1 -- 172.21.15.110:0/2396355139 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c00031940 con 0x7f6bfc0c9aa0 2024-01-27T03:43:32.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 5 monclient: authenticate success, global_id 4296 2024-01-27T03:43:32.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:32.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 1 -- 172.21.15.110:0/2396355139 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c00031cd0 con 0x7f6bfc0c9aa0 2024-01-27T03:43:32.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4040b0ab0 0x7ff404150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4040b0ab0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40414ecd0 0x7ff404150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient(hunting): picked mon.noname-b con 0x7ff40414ecd0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c089d9640 10 monclient: dump: 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.283 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.284 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.284 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.284 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 4 set_mon_vals no callback set 2024-01-27T03:43:32.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4041465d0 con 0x7ff404072100 2024-01-27T03:43:32.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4041462a0 con 0x7ff40414ecd0 2024-01-27T03:43:32.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f567f7fe640 10 monclient(hunting): get_auth_request con 0x7f5678036380 auth_method 0 2024-01-27T03:43:32.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f567f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f567f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f567f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40414aee0 con 0x7ff4040b0ab0 2024-01-27T03:43:32.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.232+0000 7ff40c11d640 10 monclient(hunting): authenticate will time out at 1495.586799s 2024-01-27T03:43:32.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 10 monclient: shutdown 2024-01-27T03:43:32.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/2396355139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bfc0c9aa0 msgr2=0x7f6bfc0c8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 1 --2- 172.21.15.110:0/2396355139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bfc0c9aa0 0x7f6bfc0c8f60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00c9f0 tx=0x7f6c0002b880 comp rx=0 tx=0).stop 2024-01-27T03:43:32.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7fd227fff640 1 -- 172.21.15.110:0/513500446 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0317e0 con 0x7fd2300b4680 2024-01-27T03:43:32.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7fd227fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 10 monclient: dump: 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/2396355139 shutdown_connections 2024-01-27T03:43:32.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 1 --2- 172.21.15.110:0/2396355139 >> v2:172.21.15.162:3300/0 conn(0x7f6bfc0c5a20 0x7f6bfc0d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.233+0000 7f6c0bc3f640 1 --2- 172.21.15.110:0/2396355139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bfc0c9aa0 0x7f6bfc0c8f60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/2396355139 >> 172.21.15.110:0/2396355139 conn(0x7f6bfc0061e0 msgr2=0x7f6bfc0065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 1 -- 172.21.15.110:0/513500446 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22c031940 con 0x7fd2300b4680 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 5 monclient: authenticate success, global_id 4299 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 1 -- 172.21.15.110:0/513500446 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c031cd0 con 0x7fd2300b4680 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd227fff640 10 monclient: dump: 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.290 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.291 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.291 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/2396355139 shutdown_connections 2024-01-27T03:43:32.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/2396355139 wait complete. 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 4 set_mon_vals no callback set 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 10 monclient: shutdown 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 1 -- 172.21.15.110:0/513500446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2300b4680 msgr2=0x7fd230142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 1 --2- 172.21.15.110:0/513500446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2300b4680 0x7fd230142970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2300b4560 tx=0x7fd22c02b880 comp rx=0 tx=0).stop 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 1 -- 172.21.15.110:0/513500446 shutdown_connections 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 1 --2- 172.21.15.110:0/513500446 >> v2:172.21.15.162:3300/0 conn(0x7fd230142eb0 0x7fd2300b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 1 --2- 172.21.15.110:0/513500446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300b4a60 0x7fd2300b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 1 --2- 172.21.15.110:0/513500446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2300b4680 0x7fd230142970 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 1 -- 172.21.15.110:0/513500446 >> 172.21.15.110:0/513500446 conn(0x7fd230066490 msgr2=0x7fd2300668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 1 -- 172.21.15.110:0/513500446 shutdown_connections 2024-01-27T03:43:32.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.234+0000 7fd23733e640 1 -- 172.21.15.110:0/513500446 wait complete. 2024-01-27T03:43:32.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f6c0bc3f640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f6c0bc3f640 10 monclient: monmap: 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f6c0bc3f640 1 librados: starting msgr at 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f6c0bc3f640 1 librados: starting objecter 2024-01-27T03:43:32.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f6c0bc3f640 1 Processor -- start 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f6c0bc3f640 1 -- start start 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f6c0bc3f640 1 librados: setting wanted keys 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f6c0bc3f640 1 librados: calling monclient init 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f6c0bc3f640 10 monclient: init 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7fd23733e640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7fd23733e640 10 monclient: monmap: 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.295 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.296 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.296 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.296 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.296 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.296 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.296 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-01-27T03:43:32.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7fd23733e640 1 librados: starting msgr at 2024-01-27T03:43:32.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7fd23733e640 1 librados: starting objecter 2024-01-27T03:43:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 1 -- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78000b41f0 msgr2=0x7f78001cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78000b41f0 0x7f78001cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 1 -- 172.21.15.110:0/259790215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7800143e70 msgr2=0x7f780013de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7800143e70 0x7f780013de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f7805a92640 1 -- 172.21.15.110:0/259790215 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc0094d0 con 0x7f78000b4930 2024-01-27T03:43:32.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f567f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.235+0000 7f567f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f567f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 Processor -- start 2024-01-27T03:43:32.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 -- start start 2024-01-27T03:43:32.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 librados: setting wanted keys 2024-01-27T03:43:32.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 librados: calling monclient init 2024-01-27T03:43:32.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient: init 2024-01-27T03:43:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T03:43:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bfc0c32d0 0x7f6bfc0c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient(hunting): picked mon.noname-c con 0x7f6bfc0c32d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bfc0c5a20 0x7f6bfc02c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient(hunting): picked mon.noname-a con 0x7f6bfc0c5a20 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc0c9aa0 0x7f6bfc02cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient(hunting): picked mon.noname-b con 0x7f6bfc0c9aa0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc02bc80 con 0x7f6bfc0c5a20 2024-01-27T03:43:32.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc0c8ab0 con 0x7f6bfc0c9aa0 2024-01-27T03:43:32.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc0c8f40 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7f6c0bc3f640 10 monclient(hunting): authenticate will time out at 1495.590315s 2024-01-27T03:43:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:43:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2300b4680 0x7fd2301459f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2300b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd230142eb0 0x7fd2300ab3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient(hunting): picked mon.noname-b con 0x7fd230142eb0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300ad960 0x7fd23013b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2300ad960 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300adf80 con 0x7fd2300b4680 2024-01-27T03:43:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300ae410 con 0x7fd230142eb0 2024-01-27T03:43:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300ad540 con 0x7fd2300ad960 2024-01-27T03:43:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.236+0000 7fd23733e640 10 monclient(hunting): authenticate will time out at 1495.590725s 2024-01-27T03:43:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7ff4096b8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4040b0ab0 0x7ff404150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7ff40a6ba640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40414ecd0 0x7ff404150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 1 -- 172.21.15.110:0/3310602383 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190031830 con 0x7f1194072a50 2024-01-27T03:43:32.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 10 monclient: dump: 2024-01-27T03:43:32.304 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.304 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 1 -- 172.21.15.110:0/3310602383 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1190031990 con 0x7f1194072a50 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.238+0000 7f1199296640 1 -- 172.21.15.110:0/3310602383 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190031d20 con 0x7f1194072a50 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f1199296640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f1199296640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f1199296640 10 monclient: dump: 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 5 monclient: authenticate success, global_id 4294 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 4 set_mon_vals no callback set 2024-01-27T03:43:32.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f7805a92640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f780013d8d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f77f800c7f0 tx=0x7f77f800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f77f6ffd640 10 client.?.objecter ms_handle_connect 0x7f78000b4930 2024-01-27T03:43:32.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f77f6ffd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f567dffb640 1 -- 172.21.15.110:0/2408442023 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207711111 0 0) 0x7f56780373a0 con 0x7f5678034d00 2024-01-27T03:43:32.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f567dffb640 10 monclient(hunting): _negotiate 2024-01-27T03:43:32.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f567dffb640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f567dffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f567dffb640 10 monclient(hunting): my global_id is 4300 2024-01-27T03:43:32.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f567dffb640 1 -- 172.21.15.110:0/2408442023 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5668003520 con 0x7f5678034d00 2024-01-27T03:43:32.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f6c099db640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bfc0c32d0 0x7f6bfc0c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 10 monclient: shutdown 2024-01-27T03:43:32.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 1 -- 172.21.15.110:0/3310602383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1194072a50 msgr2=0x7f11940aed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 1 --2- 172.21.15.110:0/3310602383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1194072a50 0x7f11940aed60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f11900097a0 tx=0x7f119002b8d0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 1 -- 172.21.15.110:0/3310602383 shutdown_connections 2024-01-27T03:43:32.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 1 --2- 172.21.15.110:0/3310602383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11940acef0 0x7f11940ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 1 --2- 172.21.15.110:0/3310602383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1194072a50 0x7f11940aed60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 1 -- 172.21.15.110:0/3310602383 >> 172.21.15.110:0/3310602383 conn(0x7f11940b3b70 msgr2=0x7f11940b3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 1 -- 172.21.15.110:0/3310602383 shutdown_connections 2024-01-27T03:43:32.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.239+0000 7f119b29a640 1 -- 172.21.15.110:0/3310602383 wait complete. 2024-01-27T03:43:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff40a6ba640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40414ecd0 0x7ff404150dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40622/0 (socket says 172.21.15.110:40622) 2024-01-27T03:43:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff40a6ba640 1 -- 172.21.15.110:0/598992525 learned_addr learned my addr 172.21.15.110:0/598992525 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff40a6ba640 10 monclient(hunting): get_auth_request con 0x7ff40414ecd0 auth_method 0 2024-01-27T03:43:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff40a6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff40a6ba640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff40a6ba640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7f6c099db640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bfc0c32d0 0x7f6bfc0c7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38358/0 (socket says 172.21.15.110:38358) 2024-01-27T03:43:32.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7f6c099db640 1 -- 172.21.15.110:0/425095038 learned_addr learned my addr 172.21.15.110:0/425095038 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7f6c099db640 10 monclient(hunting): get_auth_request con 0x7f6bfc0c32d0 auth_method 0 2024-01-27T03:43:32.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7f6c099db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7f6c099db640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7f6c099db640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7f05627fc640 1 --2- 172.21.15.110:0/3363262173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f055c0073c0 0x7f055c0c5a80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f055c035810 tx=0x7f055802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8485660c11c9f6f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff4096b8640 10 monclient(hunting): get_auth_request con 0x7ff4040b0ab0 auth_method 0 2024-01-27T03:43:32.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff4096b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7f119b29a640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff4096b8640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7ff4096b8640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.240+0000 7f119b29a640 10 monclient: monmap: 2024-01-27T03:43:32.313 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.313 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.313 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.313 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.313 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.313 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.313 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.313 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.314 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.314 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 librados: starting msgr at 2024-01-27T03:43:32.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 librados: starting objecter 2024-01-27T03:43:32.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 Processor -- start 2024-01-27T03:43:32.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 -- start start 2024-01-27T03:43:32.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 librados: setting wanted keys 2024-01-27T03:43:32.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 librados: calling monclient init 2024-01-27T03:43:32.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient: init 2024-01-27T03:43:32.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7ff4096b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7ff4096b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7ff4096b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f6c099db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f6c099db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f6c099db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T03:43:32.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1194072a50 0x7f119413bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient(hunting): picked mon.noname-c con 0x7f1194072a50 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11940acef0 0x7f1194138070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient(hunting): picked mon.noname-b con 0x7f11940acef0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11940ae150 0x7f11941da750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient(hunting): picked mon.noname-a con 0x7f11940ae150 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194072850 con 0x7f11940ae150 2024-01-27T03:43:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119413b1e0 con 0x7f11940acef0 2024-01-27T03:43:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11941375d0 con 0x7f1194072a50 2024-01-27T03:43:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.241+0000 7f119b29a640 10 monclient(hunting): authenticate will time out at 1495.595839s 2024-01-27T03:43:32.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.242+0000 7f567f7fe640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2024-01-27T03:43:32.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.242+0000 7f567f7fe640 1 -- 172.21.15.110:0/2408442023 >> v1:172.21.15.162:6789/0 conn(0x7f5678034d00 legacy=0x7f5678035c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.242+0000 7f567f7fe640 1 -- 172.21.15.110:0/2408442023 >> v1:172.21.15.110:6789/0 conn(0x7f5678039020 legacy=0x7f56780345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.242+0000 7f567f7fe640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.242+0000 7f567f7fe640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.242+0000 7f567f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.242+0000 7f567f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.242+0000 7f567f7fe640 1 -- 172.21.15.110:0/2408442023 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5678037680 con 0x7f5678036380 2024-01-27T03:43:32.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.243+0000 7ff40a6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.243+0000 7ff40a6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.243+0000 7ff40a6ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.244+0000 7fd2358db640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300ad960 0x7fd23013b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.245+0000 7ff409eb9640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404072100 0x7ff404145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff409eb9640 10 monclient(hunting): get_auth_request con 0x7ff404072100 auth_method 0 2024-01-27T03:43:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff409eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff409eb9640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7f6c091da640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bfc0c5a20 0x7f6bfc02c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff409eb9640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-01-27T03:43:32.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 1 -- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40414ecd0 msgr2=0x7ff404150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40414ecd0 0x7ff404150dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 1 -- 172.21.15.110:0/598992525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404072100 msgr2=0x7ff404145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404072100 0x7ff404145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.246+0000 7ff4096b8640 1 -- 172.21.15.110:0/598992525 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f40094d0 con 0x7ff4040b0ab0 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.247+0000 7f6c091da640 10 monclient(hunting): get_auth_request con 0x7f6bfc0c5a20 auth_method 0 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.247+0000 7f6c091da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.247+0000 7f6c091da640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.247+0000 7f6c091da640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.248+0000 7f6c091da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.248+0000 7f6c091da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.248+0000 7f6c091da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 1 -- 172.21.15.110:0/425095038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc0c9aa0 msgr2=0x7f6bfc02cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bfc0c9aa0 0x7f6bfc02cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 1 -- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bfc0c5a20 msgr2=0x7f6bfc02c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bfc0c5a20 0x7f6bfc02c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.249+0000 7f6c099db640 1 -- 172.21.15.110:0/425095038 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c000094d0 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.251+0000 7fd2358db640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300ad960 0x7fd23013b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38368/0 (socket says 172.21.15.110:38368) 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.251+0000 7fd2358db640 1 -- 172.21.15.110:0/917050832 learned_addr learned my addr 172.21.15.110:0/917050832 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.251+0000 7fd2358db640 10 monclient(hunting): get_auth_request con 0x7fd2300ad960 auth_method 0 2024-01-27T03:43:32.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.251+0000 7fd2358db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.251+0000 7fd2358db640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.251+0000 7fd2358db640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7fd2350da640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2300b4680 0x7fd2301459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 1 -- 172.21.15.110:0/3363262173 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0558031870 con 0x7f055c0073c0 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 10 monclient: dump: 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 1 -- 172.21.15.110:0/3363262173 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05580319d0 con 0x7f055c0073c0 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 1 -- 172.21.15.110:0/3363262173 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0558031d60 con 0x7f055c0073c0 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0561ffb640 10 monclient: dump: 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0563fff640 5 monclient: authenticate success, global_id 4303 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0563fff640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.252+0000 7f0563fff640 4 set_mon_vals no callback set 2024-01-27T03:43:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7fd2350da640 10 monclient(hunting): get_auth_request con 0x7fd2300b4680 auth_method 0 2024-01-27T03:43:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7fd2350da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7fd2350da640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7fd2350da640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7fd2358db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7fd2358db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7fd2358db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 10 monclient: shutdown 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 1 -- 172.21.15.110:0/3363262173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f055c0073c0 msgr2=0x7f055c0c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 1 --2- 172.21.15.110:0/3363262173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f055c0073c0 0x7f055c0c5a80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f055c035810 tx=0x7f055802b910 comp rx=0 tx=0).stop 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 1 -- 172.21.15.110:0/3363262173 shutdown_connections 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 1 --2- 172.21.15.110:0/3363262173 >> v2:172.21.15.110:3300/0 conn(0x7f055c031740 0x7f055c0d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 1 --2- 172.21.15.110:0/3363262173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f055c0073c0 0x7f055c0c5a80 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f055c035810 tx=0x7f055802b910 comp rx=0 tx=0).stop 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 1 -- 172.21.15.110:0/3363262173 >> 172.21.15.110:0/3363262173 conn(0x7f055c0061a0 msgr2=0x7f055c0065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 1 -- 172.21.15.110:0/3363262173 shutdown_connections 2024-01-27T03:43:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7f0563fff640 1 -- 172.21.15.110:0/3363262173 wait complete. 2024-01-27T03:43:32.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7fd2348d9640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd230142eb0 0x7fd2300ab3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.253+0000 7fd2350da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.254+0000 7fd2350da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.254+0000 7fd2350da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.254+0000 7f0563fff640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.254+0000 7f0563fff640 10 monclient: monmap: 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.254+0000 7f0563fff640 1 librados: starting msgr at 2024-01-27T03:43:32.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.254+0000 7f0563fff640 1 librados: starting objecter 2024-01-27T03:43:32.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.255+0000 7f567f7fe640 1 --2- 172.21.15.110:0/2408442023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5678036380 0x7f56780d7f20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f56700097a0 tx=0x7f567002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46148e50c6c8757 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.255+0000 7f0563fff640 1 Processor -- start 2024-01-27T03:43:32.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.255+0000 7f0563fff640 1 -- start start 2024-01-27T03:43:32.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.255+0000 7f0563fff640 1 librados: setting wanted keys 2024-01-27T03:43:32.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.255+0000 7f0563fff640 1 librados: calling monclient init 2024-01-27T03:43:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.255+0000 7f0563fff640 10 monclient: init 2024-01-27T03:43:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.255+0000 7ff4096b8640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4040b0ab0 0x7ff404150890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4033000 tx=0x7ff3f40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.255+0000 7ff3fa7fc640 10 client.?.objecter ms_handle_connect 0x7ff4040b0ab0 2024-01-27T03:43:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.255+0000 7ff3fa7fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:43:32.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f055c006fe0 0x7f055c0c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient(hunting): picked mon.noname-b con 0x7f055c006fe0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f055c031740 0x7f055c0c0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient(hunting): picked mon.noname-c con 0x7f055c031740 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f055c0c6da0 0x7f055c0c0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient(hunting): picked mon.noname-a con 0x7f055c0c6da0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055c0c12c0 con 0x7f055c0c6da0 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055c0c9b80 con 0x7f055c006fe0 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055c0c97e0 con 0x7f055c031740 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7f0563fff640 10 monclient(hunting): authenticate will time out at 1495.610195s 2024-01-27T03:43:32.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7fd2348d9640 10 monclient(hunting): get_auth_request con 0x7fd230142eb0 auth_method 0 2024-01-27T03:43:32.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7fd2348d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7fd2348d9640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.256+0000 7fd2348d9640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 1 -- 172.21.15.110:0/917050832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd230142eb0 msgr2=0x7fd2300ab3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd230142eb0 0x7fd2300ab3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 1 -- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2300b4680 msgr2=0x7fd2301459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2300b4680 0x7fd2301459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.258+0000 7fd2358db640 1 -- 172.21.15.110:0/917050832 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0094d0 con 0x7fd2300ad960 2024-01-27T03:43:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 1 -- 172.21.15.110:0/3514347205 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0317c0 con 0x7ff9f80b4a60 2024-01-27T03:43:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 10 monclient: dump: 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 1 -- 172.21.15.110:0/3514347205 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec031920 con 0x7ff9f80b4a60 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9fed9e640 5 monclient: authenticate success, global_id 4293 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9fed9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9fed9e640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9fed9e640 4 set_mon_vals no callback set 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 1 -- 172.21.15.110:0/3514347205 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec031cb0 con 0x7ff9f80b4a60 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.261+0000 7ff9f77fe640 10 monclient: dump: 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.341 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 10 monclient: shutdown 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3514347205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9f80b4a60 msgr2=0x7ff9f80b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 1 --2- 172.21.15.110:0/3514347205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9f80b4a60 0x7ff9f80b1470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff9f80b4560 tx=0x7ff9ec02b860 comp rx=0 tx=0).stop 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3514347205 shutdown_connections 2024-01-27T03:43:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 1 --2- 172.21.15.110:0/3514347205 >> v2:172.21.15.162:3300/0 conn(0x7ff9f8142eb0 0x7ff9f80b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 1 --2- 172.21.15.110:0/3514347205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9f80b4a60 0x7ff9f80b1470 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 1 --2- 172.21.15.110:0/3514347205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9f80b4680 0x7ff9f8142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3514347205 >> 172.21.15.110:0/3514347205 conn(0x7ff9f8066490 msgr2=0x7ff9f80668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3514347205 shutdown_connections 2024-01-27T03:43:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3514347205 wait complete. 2024-01-27T03:43:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7f6c099db640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bfc0c32d0 0x7f6bfc0c7e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400c410 tx=0x7f6bf400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7f6bfa7fc640 10 client.?.objecter ms_handle_connect 0x7f6bfc0c32d0 2024-01-27T03:43:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7f6bfa7fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7ff9fed9e640 10 monclient: monmap: 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.344 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.345 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.262+0000 7f119aa99640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11940ae150 0x7f11941da750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7ff9fed9e640 1 librados: starting msgr at 2024-01-27T03:43:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7ff9fed9e640 1 librados: starting objecter 2024-01-27T03:43:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7ff9fed9e640 1 Processor -- start 2024-01-27T03:43:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7ff9fed9e640 1 -- start start 2024-01-27T03:43:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7ff9fed9e640 1 librados: setting wanted keys 2024-01-27T03:43:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7ff9fed9e640 1 librados: calling monclient init 2024-01-27T03:43:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7ff9fed9e640 10 monclient: init 2024-01-27T03:43:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7f119aa99640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11940ae150 0x7f11941da750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50050/0 (socket says 172.21.15.110:50050) 2024-01-27T03:43:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7f119aa99640 1 -- 172.21.15.110:0/2671337598 learned_addr learned my addr 172.21.15.110:0/2671337598 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7ff9fed9e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7ff9fed9e640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:43:32.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7f119aa99640 10 monclient(hunting): get_auth_request con 0x7f11940ae150 auth_method 0 2024-01-27T03:43:32.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7f119aa99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7f119aa99640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.263+0000 7f119aa99640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9f80b4680 0x7ff9f80ac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f80b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9f8142eb0 0x7ff9f80aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f8142eb0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9f8070b00 0x7ff9f80aa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 10 monclient(hunting): picked mon.noname-c con 0x7ff9f8070b00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80abdd0 con 0x7ff9f80b4680 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80ac0b0 con 0x7ff9f8142eb0 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80aa8e0 con 0x7ff9f8070b00 2024-01-27T03:43:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7ff9fed9e640 10 monclient(hunting): authenticate will time out at 1495.618041s 2024-01-27T03:43:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.264+0000 7f05637fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f055c0c6da0 0x7f055c0c0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7f1199a97640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11940acef0 0x7f1194138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7f119aa99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7f119aa99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7f119aa99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7fd2358db640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300ad960 0x7fd23013b220 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd22800b9e0 tx=0x7fd22800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7fd225ffb640 10 client.?.objecter ms_handle_connect 0x7fd2300ad960 2024-01-27T03:43:32.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7fd225ffb640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7f1199a97640 10 monclient(hunting): get_auth_request con 0x7f11940acef0 auth_method 0 2024-01-27T03:43:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7f1199a97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7f1199a97640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.265+0000 7f1199a97640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7ff9f7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9f8142eb0 0x7ff9f80aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7f05637fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f055c0c6da0 0x7f055c0c0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50060/0 (socket says 172.21.15.110:50060) 2024-01-27T03:43:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7f05637fe640 1 -- 172.21.15.110:0/1693867931 learned_addr learned my addr 172.21.15.110:0/1693867931 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7f05637fe640 10 monclient(hunting): get_auth_request con 0x7f055c0c6da0 auth_method 0 2024-01-27T03:43:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7f05637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7f05637fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7f05637fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7f1199a97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7f1199a97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.266+0000 7f1199a97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.267+0000 7ff9f7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9f8142eb0 0x7ff9f80aca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40674/0 (socket says 172.21.15.110:40674) 2024-01-27T03:43:32.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.267+0000 7ff9f7fff640 1 -- 172.21.15.110:0/3972849549 learned_addr learned my addr 172.21.15.110:0/3972849549 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.267+0000 7ff9f7fff640 10 monclient(hunting): get_auth_request con 0x7ff9f8142eb0 auth_method 0 2024-01-27T03:43:32.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.267+0000 7ff9f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.267+0000 7ff9f7fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.267+0000 7ff9f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 1 -- 172.21.15.110:0/2408442023 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670031830 con 0x7f5678036380 2024-01-27T03:43:32.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 10 monclient: dump: 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 1 -- 172.21.15.110:0/2408442023 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5670031990 con 0x7f5678036380 2024-01-27T03:43:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 1 -- 172.21.15.110:0/2408442023 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670031d20 con 0x7f5678036380 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567dffb640 10 monclient: dump: 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567ffff640 5 monclient: authenticate success, global_id 4306 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567ffff640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.268+0000 7f567ffff640 4 set_mon_vals no callback set 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 10 monclient: shutdown 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 1 -- 172.21.15.110:0/2408442023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5678036380 msgr2=0x7f56780d7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 1 --2- 172.21.15.110:0/2408442023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5678036380 0x7f56780d7f20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f56700097a0 tx=0x7f567002b8d0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 1 -- 172.21.15.110:0/2408442023 shutdown_connections 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 1 --2- 172.21.15.110:0/2408442023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5678036380 0x7f56780d7f20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 1 -- 172.21.15.110:0/2408442023 >> 172.21.15.110:0/2408442023 conn(0x7f56780061a0 msgr2=0x7f5678006580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 1 -- 172.21.15.110:0/2408442023 shutdown_connections 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 1 -- 172.21.15.110:0/2408442023 wait complete. 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.269+0000 7f567ffff640 10 monclient: monmap: 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-01-27T03:43:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 1 -- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1194072a50 msgr2=0x7f119413bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1194072a50 0x7f119413bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 1 -- 172.21.15.110:0/2671337598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11940acef0 msgr2=0x7f1194138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f567ffff640 1 librados: starting msgr at 2024-01-27T03:43:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11940acef0 0x7f1194138070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 1 monclient: found mon.noname-a 2024-01-27T03:43:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f119aa99640 1 -- 172.21.15.110:0/2671337598 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1190009400 con 0x7f11940ae150 2024-01-27T03:43:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f567ffff640 1 librados: starting objecter 2024-01-27T03:43:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f567ffff640 1 Processor -- start 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 1 -- 172.21.15.110:0/598992525 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4032920 con 0x7ff4040b0ab0 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 10 monclient: dump: 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.361 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 1 -- 172.21.15.110:0/598992525 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f4032aa0 con 0x7ff4040b0ab0 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 1 -- 172.21.15.110:0/598992525 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4032e50 con 0x7ff4040b0ab0 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fa7fc640 10 monclient: dump: 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f05637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f05637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f05637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff40c11d640 5 monclient: authenticate success, global_id 4277 2024-01-27T03:43:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff40c11d640 10 monclient: _renew_subs 2024-01-27T03:43:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f567ffff640 1 -- start start 2024-01-27T03:43:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f567ffff640 1 librados: setting wanted keys 2024-01-27T03:43:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f567ffff640 1 librados: calling monclient init 2024-01-27T03:43:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7f567ffff640 10 monclient: init 2024-01-27T03:43:32.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff3fbfff640 4 set_mon_vals no callback set 2024-01-27T03:43:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff40c11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4041459c0 con 0x7ff4040b0ab0 2024-01-27T03:43:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff40c11d640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff40c11d640 10 monclient: _renew_subs 2024-01-27T03:43:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff40c11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4041514f0 con 0x7ff4040b0ab0 2024-01-27T03:43:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.270+0000 7ff40c11d640 1 librados: init done 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5678034d00 0x7f5678155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5678034d00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5678036380 0x7f56780c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f5678036380 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5678039020 0x7f56780c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f5678039020 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5678156890 con 0x7f5678034d00 2024-01-27T03:43:32.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56780c6850 con 0x7f5678039020 2024-01-27T03:43:32.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5678156320 con 0x7f5678036380 2024-01-27T03:43:32.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f567ffff640 10 monclient(hunting): authenticate will time out at 1495.625205s 2024-01-27T03:43:32.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7ff40c11d640 10 client.4277.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7ff40c11d640 15 client.4277.objecter _session_command_op_assign -1 1 2024-01-27T03:43:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7ff40c11d640 20 client.4277.objecter _get_session osd=-1 returning homeless 2024-01-27T03:43:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7ff40c11d640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7ff40c11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff4040b0600 con 0x7ff4040b0ab0 2024-01-27T03:43:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7ff9f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7ff9f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7ff9f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.271+0000 7f0562ffd640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f055c006fe0 0x7f055c0c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-01-27T03:43:32.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 1 -- 172.21.15.110:0/1693867931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f055c031740 msgr2=0x7f055c0c0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f055c031740 0x7f055c0c0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 1 -- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f055c006fe0 msgr2=0x7f055c0c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f055c006fe0 0x7f055c0c7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 1 monclient: found mon.noname-a 2024-01-27T03:43:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f05637fe640 1 -- 172.21.15.110:0/1693867931 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05580094d0 con 0x7f055c0c6da0 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 1 -- 172.21.15.110:0/259790215 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f8003d80 con 0x7f78000b4930 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 10 monclient: dump: 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.372 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 1 -- 172.21.15.110:0/259790215 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f8005070 con 0x7f78000b4930 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 1 -- 172.21.15.110:0/259790215 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f8005460 con 0x7f78000b4930 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f77f6ffd640 10 monclient: dump: 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.373 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f78084f7640 5 monclient: authenticate success, global_id 4274 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f78084f7640 10 monclient: _renew_subs 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f7804a90640 4 set_mon_vals no callback set 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f78084f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78000afdf0 con 0x7f78000b4930 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f78084f7640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f78084f7640 10 monclient: _renew_subs 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f78084f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780014a950 con 0x7f78000b4930 2024-01-27T03:43:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.273+0000 7f78084f7640 1 librados: init done 2024-01-27T03:43:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.274+0000 7f78084f7640 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.274+0000 7f78084f7640 15 client.4274.objecter _session_command_op_assign -1 1 2024-01-27T03:43:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.274+0000 7f78084f7640 20 client.4274.objecter _get_session osd=-1 returning homeless 2024-01-27T03:43:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.274+0000 7f78084f7640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.274+0000 7f78084f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.274+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78000b1910 con 0x7f78000b4930 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.274+0000 7ff9fcb3a640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9f80b4680 0x7ff9f80ac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 1 -- 172.21.15.110:0/3972849549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9f8070b00 msgr2=0x7ff9f80aa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9f8070b00 0x7ff9f80aa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 1 -- 172.21.15.110:0/3972849549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9f80b4680 msgr2=0x7ff9f80ac4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9f80b4680 0x7ff9f80ac4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 1 monclient: found mon.noname-b 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.275+0000 7ff9f7fff640 1 -- 172.21.15.110:0/3972849549 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec0094d0 con 0x7ff9f8142eb0 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.279+0000 7ff9f7fff640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9f8142eb0 0x7ff9f80aca10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec000c00 tx=0x7ff9ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.279+0000 7ff9f57fa640 10 client.?.objecter ms_handle_connect 0x7ff9f8142eb0 2024-01-27T03:43:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.279+0000 7ff9f57fa640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.281+0000 7f05637fe640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f055c0c6da0 0x7f055c0c0d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0564073520 tx=0x7f0564068370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.281+0000 7f054b7fe640 10 client.?.objecter ms_handle_connect 0x7f055c0c6da0 2024-01-27T03:43:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.281+0000 7f054b7fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.281+0000 7f119aa99640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11940ae150 0x7f11941da750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f118c00efc0 tx=0x7f118c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.281+0000 7f118affd640 10 client.?.objecter ms_handle_connect 0x7f11940ae150 2024-01-27T03:43:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.281+0000 7f118affd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.284+0000 7f567f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5678039020 0x7f56780c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 1 -- 172.21.15.110:0/3972849549 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec032860 con 0x7ff9f8142eb0 2024-01-27T03:43:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 10 monclient: dump: 2024-01-27T03:43:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 1 -- 172.21.15.110:0/3972849549 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec0329c0 con 0x7ff9f8142eb0 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9fed9e640 5 monclient: authenticate success, global_id 4318 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9fed9e640 10 monclient: _renew_subs 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9fed9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f80aab00 con 0x7ff9f8142eb0 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9fed9e640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f67fc640 4 set_mon_vals no callback set 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9fed9e640 10 monclient: _renew_subs 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 1 -- 172.21.15.110:0/3972849549 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec032d70 con 0x7ff9f8142eb0 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9fed9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f812fb40 con 0x7ff9f8142eb0 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9fed9e640 1 librados: init done 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7ff9f57fa640 10 monclient: dump: 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.382 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7f567f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5678039020 0x7f56780c6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40690/0 (socket says 172.21.15.110:40690) 2024-01-27T03:43:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7f567f7fe640 1 -- 172.21.15.110:0/3098588689 learned_addr learned my addr 172.21.15.110:0/3098588689 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7f567f7fe640 10 monclient(hunting): get_auth_request con 0x7f5678039020 auth_method 0 2024-01-27T03:43:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7f567f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7f567f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7f567f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 1 -- 172.21.15.110:0/917050832 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22800cb20 con 0x7fd2300ad960 2024-01-27T03:43:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 10 monclient: dump: 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 1 -- 172.21.15.110:0/917050832 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22800cca0 con 0x7fd2300ad960 2024-01-27T03:43:32.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 1 -- 172.21.15.110:0/917050832 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd228014680 con 0x7fd2300ad960 2024-01-27T03:43:32.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:32.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7f567effd640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5678034d00 0x7f5678155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd225ffb640 10 monclient: dump: 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd2277fe640 4 set_mon_vals no callback set 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd23733e640 5 monclient: authenticate success, global_id 4283 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd23733e640 10 monclient: _renew_subs 2024-01-27T03:43:32.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd23733e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.285+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23013b760 con 0x7fd2300ad960 2024-01-27T03:43:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7fd23733e640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7fd23733e640 10 monclient: _renew_subs 2024-01-27T03:43:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7fd23733e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23013bc20 con 0x7fd2300ad960 2024-01-27T03:43:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7fd23733e640 1 librados: init done 2024-01-27T03:43:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7f567f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7f567f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7f567f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7ff9fed9e640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7ff9fed9e640 15 client.4318.objecter _session_command_op_assign -1 1 2024-01-27T03:43:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7ff9fed9e640 20 client.4318.objecter _get_session osd=-1 returning homeless 2024-01-27T03:43:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7ff9fed9e640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7ff9fed9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.286+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff9f80abf40 con 0x7ff9f8142eb0 2024-01-27T03:43:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.287+0000 7fd23733e640 10 client.4283.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.287+0000 7fd23733e640 15 client.4283.objecter _session_command_op_assign -1 1 2024-01-27T03:43:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.287+0000 7fd23733e640 20 client.4283.objecter _get_session osd=-1 returning homeless 2024-01-27T03:43:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.287+0000 7fd23733e640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.287+0000 7fd23733e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.287+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2300af7e0 con 0x7fd2300ad960 2024-01-27T03:43:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.288+0000 7f567effd640 10 monclient(hunting): get_auth_request con 0x7f5678034d00 auth_method 0 2024-01-27T03:43:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.288+0000 7f567effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.288+0000 7f567effd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.288+0000 7f567effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f118affd640 1 -- 172.21.15.110:0/2671337598 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c00ced0 con 0x7f11940ae150 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f118affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f118affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f118affd640 10 monclient: dump: 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.391 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f118affd640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f118affd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f118affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f119b29a640 5 monclient: authenticate success, global_id 4317 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f119b29a640 10 monclient: _renew_subs 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f119b29a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119413ad50 con 0x7f11940ae150 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f118affd640 1 -- 172.21.15.110:0/2671337598 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c00f040 con 0x7f11940ae150 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f118affd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f119b29a640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.289+0000 7f1198a95640 4 set_mon_vals no callback set 2024-01-27T03:43:32.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f567effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f567effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f567effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f119b29a640 10 monclient: _renew_subs 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f119b29a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11941dac90 con 0x7f11940ae150 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f119b29a640 1 librados: init done 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f118affd640 1 -- 172.21.15.110:0/2671337598 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c0146c0 con 0x7f11940ae150 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f118affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f118affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.290+0000 7f118affd640 10 monclient: dump: 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.291+0000 7f119b29a640 10 client.4317.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.291+0000 7f119b29a640 15 client.4317.objecter _session_command_op_assign -1 1 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.291+0000 7f119b29a640 20 client.4317.objecter _get_session osd=-1 returning homeless 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.291+0000 7f119b29a640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.291+0000 7f119b29a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.291+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1194075e20 con 0x7f11940ae150 2024-01-27T03:43:32.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.299+0000 7ff3fa7fc640 1 -- 172.21.15.110:0/598992525 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7ff3f404e440 con 0x7ff4040b0ab0 2024-01-27T03:43:32.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7ff3fa7fc640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff3e806c9d0 0x7ff3e806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-01-27T03:43:32.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 1 -- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5678036380 msgr2=0x7f56780c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5678036380 0x7f56780c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 1 -- 172.21.15.110:0/3098588689 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5678039020 msgr2=0x7f56780c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5678039020 0x7f56780c6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 1 monclient: found mon.noname-a 2024-01-27T03:43:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.300+0000 7f567effd640 1 -- 172.21.15.110:0/3098588689 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5670009400 con 0x7f5678034d00 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 1 -- 172.21.15.110:0/1693867931 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056404efc0 con 0x7f055c0c6da0 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 10 monclient: dump: 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 1 -- 172.21.15.110:0/1693867931 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0564067700 con 0x7f055c0c6da0 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 1 -- 172.21.15.110:0/1693867931 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564067af0 con 0x7f055c0c6da0 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f0563fff640 5 monclient: authenticate success, global_id 4320 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f0563fff640 10 monclient: _renew_subs 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f0563fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055c0c8070 con 0x7f055c0c6da0 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f05617fa640 4 set_mon_vals no callback set 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f054b7fe640 10 monclient: dump: 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f0563fff640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f0563fff640 10 monclient: _renew_subs 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f0563fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055c160d70 con 0x7f055c0c6da0 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.301+0000 7f0563fff640 1 librados: init done 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.302+0000 7f0563fff640 10 client.4320.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.302+0000 7f0563fff640 15 client.4320.objecter _session_command_op_assign -1 1 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.302+0000 7f0563fff640 20 client.4320.objecter _get_session osd=-1 returning homeless 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.302+0000 7f0563fff640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.302+0000 7f0563fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.302+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f055c0378c0 con 0x7f055c0c6da0 2024-01-27T03:43:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.306+0000 7ff409eb9640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff3e806c9d0 0x7ff3e806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.306+0000 7ff409eb9640 10 monclient: get_auth_request con 0x7ff3e806c9d0 auth_method 0 2024-01-27T03:43:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.306+0000 7ff409eb9640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff3e806c9d0 0x7ff3e806ee90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff400007fd0 tx=0x7ff400009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.306+0000 7ff3fa7fc640 10 client.4277.objecter ms_handle_connect 0x7ff3e806c9d0 2024-01-27T03:43:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.309+0000 7ff9f57fa640 1 -- 172.21.15.110:0/3972849549 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7ff9ec04d3f0 con 0x7ff9f8142eb0 2024-01-27T03:43:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.309+0000 7ff9f57fa640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff9e406c9f0 0x7ff9e406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.309+0000 7ff9fcb3a640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff9e406c9f0 0x7ff9e406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.309+0000 7ff9fcb3a640 10 monclient: get_auth_request con 0x7ff9e406c9f0 auth_method 0 2024-01-27T03:43:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.310+0000 7ff9f57fa640 10 client.4318.objecter ms_handle_connect 0x7ff9e406c9f0 2024-01-27T03:43:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.310+0000 7ff9fcb3a640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff9e406c9f0 0x7ff9e406eeb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0009fd0 tx=0x7ff9f000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 1 -- 172.21.15.110:0/598992525 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff3f407b2a0 con 0x7ff4040b0ab0 2024-01-27T03:43:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 10 client.4277.objecter ms_dispatch 0x7ff404000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 3 client.4277.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 3 client.4277.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 10 client.4277.objecter checking command 1 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7ff3e8072db0 0x7ff3e80751d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 20 client.4277.objecter _get_session s=0x7ff3e80726e0 osd=4 3 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 20 client.4277.objecter put_session s=0x7ff3e80726e0 osd=4 3 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 15 client.4277.objecter _session_command_op_remove -1 1 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 20 client.4277.objecter _get_session s=0x7ff3e80726e0 osd=4 3 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 20 client.4277.objecter get_session s=0x7ff3e80726e0 osd=4 3 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 15 client.4277.objecter _session_command_op_assign 4 1 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 20 client.4277.objecter put_session s=0x7ff3e80726e0 osd=4 4 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 10 client.4277.objecter _send_command 1 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 1 -- 172.21.15.110:0/598992525 --> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3e8075880 con 0x7ff3e8072db0 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 20 client.4277.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 1 -- 172.21.15.110:0/598992525 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3f4076e40 con 0x7ff4040b0ab0 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.314+0000 7ff3fa7fc640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-27T03:43:32.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.315+0000 7ff40a6ba640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7ff3e8072db0 0x7ff3e80751d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.315+0000 7ff40a6ba640 10 monclient: get_auth_request con 0x7ff3e8072db0 auth_method 0 2024-01-27T03:43:32.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.315+0000 7ff9f57fa640 1 -- 172.21.15.110:0/3972849549 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff9ec07af00 con 0x7ff9f8142eb0 2024-01-27T03:43:32.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.315+0000 7ff9f57fa640 10 client.4318.objecter ms_dispatch 0x7ff9f8000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.315+0000 7ff9f57fa640 3 client.4318.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.315+0000 7ff9f57fa640 3 client.4318.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.315+0000 7ff9f57fa640 10 client.4318.objecter checking command 1 2024-01-27T03:43:32.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7ff9e4072dd0 0x7ff9e40751f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 20 client.4318.objecter _get_session s=0x7ff9e4072700 osd=7 3 2024-01-27T03:43:32.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 20 client.4318.objecter put_session s=0x7ff9e4072700 osd=7 3 2024-01-27T03:43:32.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 15 client.4318.objecter _session_command_op_remove -1 1 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 20 client.4318.objecter _get_session s=0x7ff9e4072700 osd=7 3 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 20 client.4318.objecter get_session s=0x7ff9e4072700 osd=7 3 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 15 client.4318.objecter _session_command_op_assign 7 1 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 20 client.4318.objecter put_session s=0x7ff9e4072700 osd=7 4 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 10 client.4318.objecter _send_command 1 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 1 -- 172.21.15.110:0/3972849549 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9e40758a0 con 0x7ff9e4072dd0 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9f57fa640 20 client.4318.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.316+0000 7ff9fd33b640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7ff9e4072dd0 0x7ff9e40751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.317+0000 7ff9fd33b640 10 monclient: get_auth_request con 0x7ff9e4072dd0 auth_method 0 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.317+0000 7ff9f57fa640 1 -- 172.21.15.110:0/3972849549 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9ec076aa0 con 0x7ff9f8142eb0 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.317+0000 7ff9f57fa640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.319+0000 7ff9f57fa640 10 client.4318.objecter ms_handle_connect 0x7ff9e4072dd0 2024-01-27T03:43:32.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.319+0000 7ff9fd33b640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7ff9e4072dd0 0x7ff9e40751f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.320+0000 7ff40a6ba640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7ff3e8072db0 0x7ff3e80751d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.320+0000 7ff3fa7fc640 10 client.4277.objecter ms_handle_connect 0x7ff3e8072db0 2024-01-27T03:43:32.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.320+0000 7f118affd640 1 -- 172.21.15.110:0/2671337598 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f118c014820 con 0x7f11940ae150 2024-01-27T03:43:32.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.321+0000 7f118affd640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f117806c9d0 0x7f117806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.321+0000 7f119a298640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f117806c9d0 0x7f117806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.321+0000 7f119a298640 10 monclient: get_auth_request con 0x7f117806c9d0 auth_method 0 2024-01-27T03:43:32.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.322+0000 7f119a298640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f117806c9d0 0x7f117806ee90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1184009690 tx=0x7f1184006bd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.322+0000 7f118affd640 10 client.4317.objecter ms_handle_connect 0x7f117806c9d0 2024-01-27T03:43:32.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.331+0000 7f567effd640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5678034d00 0x7f5678155500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f568004f800 tx=0x7f5680068370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.331+0000 7f5667fff640 10 client.?.objecter ms_handle_connect 0x7f5678034d00 2024-01-27T03:43:32.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.331+0000 7f5667fff640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 1 -- 172.21.15.110:0/2671337598 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f118c059c00 con 0x7f11940ae150 2024-01-27T03:43:32.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 10 client.4317.objecter ms_dispatch 0x7f1194000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 3 client.4317.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 3 client.4317.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 10 client.4317.objecter checking command 1 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f1178072c40 0x7f1178075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 20 client.4317.objecter _get_session s=0x7f118c00f040 osd=3 3 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 20 client.4317.objecter put_session s=0x7f118c00f040 osd=3 3 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 15 client.4317.objecter _session_command_op_remove -1 1 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 20 client.4317.objecter _get_session s=0x7f118c00f040 osd=3 3 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 20 client.4317.objecter get_session s=0x7f118c00f040 osd=3 3 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 15 client.4317.objecter _session_command_op_assign 3 1 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 20 client.4317.objecter put_session s=0x7f118c00f040 osd=3 4 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 10 client.4317.objecter _send_command 1 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 1 -- 172.21.15.110:0/2671337598 --> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f118c00ced0 con 0x7f1178072c40 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 20 client.4317.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 1 -- 172.21.15.110:0/2671337598 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f118c0557a0 con 0x7f11940ae150 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.336+0000 7f118affd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-27T03:43:32.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.337+0000 7f1199a97640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f1178072c40 0x7f1178075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.337+0000 7f1199a97640 10 monclient: get_auth_request con 0x7f1178072c40 auth_method 0 2024-01-27T03:43:32.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.339+0000 7f118affd640 10 client.4317.objecter ms_handle_connect 0x7f1178072c40 2024-01-27T03:43:32.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.339+0000 7f1199a97640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f1178072c40 0x7f1178075060 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 1 -- 172.21.15.110:0/425095038 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4011680 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 10 monclient: dump: 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.416 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 1 -- 172.21.15.110:0/425095038 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf40117e0 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 1 -- 172.21.15.110:0/425095038 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4011bd0 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfa7fc640 10 monclient: dump: 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.417 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.340+0000 7f6bfbfff640 4 set_mon_vals no callback set 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f6c0bc3f640 5 monclient: authenticate success, global_id 4280 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f6c0bc3f640 10 monclient: _renew_subs 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f6c0bc3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bfc02c1b0 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f6c0bc3f640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f6c0bc3f640 10 monclient: _renew_subs 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f6c0bc3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bfc02d330 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f6c0bc3f640 1 librados: init done 2024-01-27T03:43:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f77f6ffd640 1 -- 172.21.15.110:0/259790215 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f77f80055c0 con 0x7f78000b4930 2024-01-27T03:43:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f77f6ffd640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f77e406c9f0 0x7f77e406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f7806293640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f77e406c9f0 0x7f77e406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.341+0000 7f7806293640 10 monclient: get_auth_request con 0x7f77e406c9f0 auth_method 0 2024-01-27T03:43:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.342+0000 7f6c0bc3f640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.342+0000 7f6c0bc3f640 15 client.4280.objecter _session_command_op_assign -1 1 2024-01-27T03:43:32.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.342+0000 7f6c0bc3f640 20 client.4280.objecter _get_session osd=-1 returning homeless 2024-01-27T03:43:32.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.342+0000 7f6c0bc3f640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.342+0000 7f6c0bc3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.342+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6bfc005c00 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.342+0000 7f77f6ffd640 10 client.4274.objecter ms_handle_connect 0x7f77e406c9f0 2024-01-27T03:43:32.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.342+0000 7f7806293640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f77e406c9f0 0x7f77e406eeb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f77fc035000 tx=0x7f77fc02e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 1 -- 172.21.15.110:0/259790215 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f77f8058760 con 0x7f78000b4930 2024-01-27T03:43:32.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 10 client.4274.objecter ms_dispatch 0x7f7800000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 3 client.4274.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 3 client.4274.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 10 client.4274.objecter checking command 1 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f77e4072dd0 0x7f77e40751f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 20 client.4274.objecter _get_session s=0x7f77e4072700 osd=5 3 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 20 client.4274.objecter put_session s=0x7f77e4072700 osd=5 3 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 15 client.4274.objecter _session_command_op_remove -1 1 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 20 client.4274.objecter _get_session s=0x7f77e4072700 osd=5 3 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 20 client.4274.objecter get_session s=0x7f77e4072700 osd=5 3 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 15 client.4274.objecter _session_command_op_assign 5 1 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 20 client.4274.objecter put_session s=0x7f77e4072700 osd=5 4 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 10 client.4274.objecter _send_command 1 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 1 -- 172.21.15.110:0/259790215 --> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77e40758a0 con 0x7f77e4072dd0 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 20 client.4274.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 1 -- 172.21.15.110:0/259790215 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77f8001820 con 0x7f78000b4930 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.348+0000 7f77f6ffd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-27T03:43:32.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.349+0000 7f7806a94640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f77e4072dd0 0x7f77e40751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.349+0000 7f7806a94640 10 monclient: get_auth_request con 0x7f77e4072dd0 auth_method 0 2024-01-27T03:43:32.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.350+0000 7f054b7fe640 1 -- 172.21.15.110:0/1693867931 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f0564067c50 con 0x7f055c0c6da0 2024-01-27T03:43:32.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.350+0000 7f054b7fe640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f055006c9f0 0x7f055006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.350+0000 7f0562ffd640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f055006c9f0 0x7f055006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.350+0000 7f0562ffd640 10 monclient: get_auth_request con 0x7f055006c9f0 auth_method 0 2024-01-27T03:43:32.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.351+0000 7f0562ffd640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f055006c9f0 0x7f055006eeb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0554000f60 tx=0x7f0554009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.351+0000 7f054b7fe640 10 client.4320.objecter ms_handle_connect 0x7f055006c9f0 2024-01-27T03:43:32.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.352+0000 7f7806a94640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f77e4072dd0 0x7f77e40751f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.352+0000 7f77f6ffd640 10 client.4274.objecter ms_handle_connect 0x7f77e4072dd0 2024-01-27T03:43:32.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 1 -- 172.21.15.110:0/1693867931 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f05640d2aa0 con 0x7f055c0c6da0 2024-01-27T03:43:32.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 10 client.4320.objecter ms_dispatch 0x7f0564000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 3 client.4320.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 3 client.4320.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 10 client.4320.objecter checking command 1 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f0550072dd0 0x7f05500751f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 20 client.4320.objecter _get_session s=0x7f0550072700 osd=6 3 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 20 client.4320.objecter put_session s=0x7f0550072700 osd=6 3 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 15 client.4320.objecter _session_command_op_remove -1 1 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 20 client.4320.objecter _get_session s=0x7f0550072700 osd=6 3 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 20 client.4320.objecter get_session s=0x7f0550072700 osd=6 3 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 15 client.4320.objecter _session_command_op_assign 6 1 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 20 client.4320.objecter put_session s=0x7f0550072700 osd=6 4 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 10 client.4320.objecter _send_command 1 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 1 -- 172.21.15.110:0/1693867931 --> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05500758a0 con 0x7f0550072dd0 2024-01-27T03:43:32.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 20 client.4320.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 1 -- 172.21.15.110:0/1693867931 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05640ce640 con 0x7f055c0c6da0 2024-01-27T03:43:32.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.358+0000 7f054b7fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-27T03:43:32.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.359+0000 7f05627fc640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f0550072dd0 0x7f05500751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.359+0000 7f05627fc640 10 monclient: get_auth_request con 0x7f0550072dd0 auth_method 0 2024-01-27T03:43:32.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 1 -- 172.21.15.110:0/3098588689 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5680067700 con 0x7f5678034d00 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 10 monclient: dump: 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.428 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 1 -- 172.21.15.110:0/3098588689 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5680067860 con 0x7f5678034d00 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 1 -- 172.21.15.110:0/3098588689 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5680067bf0 con 0x7f5678034d00 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567ffff640 5 monclient: authenticate success, global_id 4323 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567ffff640 10 monclient: _renew_subs 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5678030410 con 0x7f5678034d00 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:32.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f5667fff640 10 monclient: dump: 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567d7fa640 4 set_mon_vals no callback set 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567ffff640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567ffff640 10 monclient: _renew_subs 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5678155c20 con 0x7f5678034d00 2024-01-27T03:43:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.361+0000 7f567ffff640 1 librados: init done 2024-01-27T03:43:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.362+0000 7f567ffff640 10 client.4323.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.362+0000 7f567ffff640 15 client.4323.objecter _session_command_op_assign -1 1 2024-01-27T03:43:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.362+0000 7f567ffff640 20 client.4323.objecter _get_session osd=-1 returning homeless 2024-01-27T03:43:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.362+0000 7f567ffff640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.362+0000 7f567ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.362+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5678030a80 con 0x7f5678034d00 2024-01-27T03:43:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.364+0000 7f05627fc640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f0550072dd0 0x7f05500751f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.364+0000 7f054b7fe640 10 client.4320.objecter ms_handle_connect 0x7f0550072dd0 2024-01-27T03:43:32.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.367+0000 7ff9f57fa640 1 -- 172.21.15.110:0/3972849549 <== osd.7 v2:172.21.15.162:6818/68785612 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9e40758a0 con 0x7ff9e4072dd0 2024-01-27T03:43:32.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.367+0000 7ff9f57fa640 10 client.4318.objecter ms_dispatch 0x7ff9f8000b90 command_reply(tid 1: 0 ) v1 2024-01-27T03:43:32.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.367+0000 7ff9f57fa640 10 client.4318.objecter _finish_command 1 = system:0 2024-01-27T03:43:32.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.367+0000 7ff9f57fa640 20 client.4318.objecter put_session s=0x7ff9e4072700 osd=7 4 2024-01-27T03:43:32.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.367+0000 7ff9f57fa640 15 client.4318.objecter _session_command_op_remove 7 1 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.368+0000 7fd225ffb640 1 -- 172.21.15.110:0/917050832 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fd22801c360 con 0x7fd2300ad960 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.368+0000 7fd225ffb640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd21406ca20 0x7fd21406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.368+0000 7fd2350da640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd21406ca20 0x7fd21406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.369+0000 7fd2350da640 10 monclient: get_auth_request con 0x7fd21406ca20 auth_method 0 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.369+0000 7fd225ffb640 10 client.4283.objecter ms_handle_connect 0x7fd21406ca20 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.369+0000 7fd2350da640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd21406ca20 0x7fd21406eee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd22c036000 tx=0x7fd22c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 1 -- 172.21.15.110:0/917050832 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd2280589b0 con 0x7fd2300ad960 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 10 client.4283.objecter ms_dispatch 0x7fd230000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 3 client.4283.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 3 client.4283.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 10 client.4283.objecter checking command 1 2024-01-27T03:43:32.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7fd214072e00 0x7fd214075220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 20 client.4283.objecter _get_session s=0x7fd214072730 osd=2 3 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 20 client.4283.objecter put_session s=0x7fd214072730 osd=2 3 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 15 client.4283.objecter _session_command_op_remove -1 1 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 20 client.4283.objecter _get_session s=0x7fd214072730 osd=2 3 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 20 client.4283.objecter get_session s=0x7fd214072730 osd=2 3 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 15 client.4283.objecter _session_command_op_assign 2 1 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 20 client.4283.objecter put_session s=0x7fd214072730 osd=2 4 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 10 client.4283.objecter _send_command 1 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 1 -- 172.21.15.110:0/917050832 --> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2140758d0 con 0x7fd214072e00 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 20 client.4283.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 1 -- 172.21.15.110:0/917050832 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd228054c50 con 0x7fd2300ad960 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd225ffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.381+0000 7fd2348d9640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7fd214072e00 0x7fd214075220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.382+0000 7fd2348d9640 10 monclient: get_auth_request con 0x7fd214072e00 auth_method 0 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 15 client.4318.objecter _session_command_op_assign -1 2 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 20 client.4318.objecter _get_session s=0x7ff9e4072700 osd=7 3 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 20 client.4318.objecter put_session s=0x7ff9e4072700 osd=7 3 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 20 client.4318.objecter _get_session s=0x7ff9e4072700 osd=7 3 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 15 client.4318.objecter _session_command_op_remove -1 2 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 20 client.4318.objecter get_session s=0x7ff9e4072700 osd=7 3 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 15 client.4318.objecter _session_command_op_assign 7 2 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 20 client.4318.objecter put_session s=0x7ff9e4072700 osd=7 4 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 10 client.4318.objecter _send_command 2 2024-01-27T03:43:32.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.385+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff9f80abf40 con 0x7ff9e4072dd0 2024-01-27T03:43:32.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.386+0000 7fd2348d9640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7fd214072e00 0x7fd214075220 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.386+0000 7fd225ffb640 10 client.4283.objecter ms_handle_connect 0x7fd214072e00 2024-01-27T03:43:32.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.392+0000 7ff9f57fa640 1 -- 172.21.15.110:0/3972849549 <== osd.7 v2:172.21.15.162:6818/68785612 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff9f80abf40 con 0x7ff9e4072dd0 2024-01-27T03:43:32.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.392+0000 7ff9f57fa640 10 client.4318.objecter ms_dispatch 0x7ff9f8000b90 command_reply(tid 2: 0 ) v1 2024-01-27T03:43:32.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.392+0000 7ff9f57fa640 10 client.4318.objecter _finish_command 2 = system:0 2024-01-27T03:43:32.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.392+0000 7ff9f57fa640 20 client.4318.objecter put_session s=0x7ff9e4072700 osd=7 4 2024-01-27T03:43:32.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.392+0000 7ff9f57fa640 15 client.4318.objecter _session_command_op_remove 7 2 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 10 librados: watch_flush enter 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 10 librados: watch_flush exit 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 10 client.4318.objecter close_session for osd.7 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7ff9e4072dd0 msgr2=0x7ff9e40751f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7ff9e4072dd0 0x7ff9e40751f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 20 client.4318.objecter put_session s=0x7ff9e4072700 osd=7 1 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 20 client.4318.objecter shutdown clearing up homeless session... 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 10 client.4318.objecter successfully canceled tick 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff9e406c9f0 msgr2=0x7ff9e406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff9e406c9f0 0x7ff9e406eeb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0009fd0 tx=0x7ff9f000b040 comp rx=0 tx=0).stop 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 10 monclient: shutdown 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9f8142eb0 msgr2=0x7ff9f80aca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 --2- 172.21.15.110:0/3972849549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9f8142eb0 0x7ff9f80aca10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec000c00 tx=0x7ff9ec0046c0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fd33b640 1 -- 172.21.15.110:0/3972849549 reap_dead start 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 shutdown_connections 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 >> 172.21.15.110:0/3972849549 conn(0x7ff9f8066490 msgr2=0x7ff9f81373d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 shutdown_connections 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff3fa7fc640 1 -- 172.21.15.110:0/598992525 <== osd.4 v2:172.21.15.162:6826/1007517023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff3e8075880 con 0x7ff3e8072db0 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff3fa7fc640 10 client.4277.objecter ms_dispatch 0x7ff404000b90 command_reply(tid 1: 0 ) v1 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff3fa7fc640 10 client.4277.objecter _finish_command 1 = system:0 2024-01-27T03:43:32.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff3fa7fc640 20 client.4277.objecter put_session s=0x7ff3e80726e0 osd=4 4 2024-01-27T03:43:32.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff3fa7fc640 15 client.4277.objecter _session_command_op_remove 4 1 2024-01-27T03:43:32.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 -- 172.21.15.110:0/3972849549 wait complete. 2024-01-27T03:43:32.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.393+0000 7ff9fed9e640 1 librados: shutdown 2024-01-27T03:43:32.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.395+0000 7f5667fff640 1 -- 172.21.15.110:0/3098588689 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f568006b850 con 0x7f5678034d00 2024-01-27T03:43:32.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.396+0000 7f5667fff640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f566c06c9f0 0x7f566c06eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.396+0000 7f567e7fc640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f566c06c9f0 0x7f566c06eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.396+0000 7f567e7fc640 10 monclient: get_auth_request con 0x7f566c06c9f0 auth_method 0 2024-01-27T03:43:32.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.396+0000 7f5667fff640 10 client.4323.objecter ms_handle_connect 0x7f566c06c9f0 2024-01-27T03:43:32.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.396+0000 7f567e7fc640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f566c06c9f0 0x7f566c06eeb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f56780303c0 tx=0x7f5674007930 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.398+0000 7f118affd640 1 -- 172.21.15.110:0/2671337598 <== osd.3 v2:172.21.15.110:6800/2374111370 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f118c00ced0 con 0x7f1178072c40 2024-01-27T03:43:32.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.398+0000 7f118affd640 10 client.4317.objecter ms_dispatch 0x7f1194000b90 command_reply(tid 1: 0 ) v1 2024-01-27T03:43:32.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.398+0000 7f118affd640 10 client.4317.objecter _finish_command 1 = system:0 2024-01-27T03:43:32.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.398+0000 7f118affd640 20 client.4317.objecter put_session s=0x7f118c00f040 osd=3 4 2024-01-27T03:43:32.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.398+0000 7f118affd640 15 client.4317.objecter _session_command_op_remove 3 1 2024-01-27T03:43:32.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 1 -- 172.21.15.110:0/3098588689 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f56800d2960 con 0x7f5678034d00 2024-01-27T03:43:32.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 10 client.4323.objecter ms_dispatch 0x7f5680000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 3 client.4323.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 3 client.4323.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 10 client.4323.objecter checking command 1 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7f566c072dd0 0x7f566c0751f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 20 client.4323.objecter _get_session s=0x7f566c072700 osd=0 3 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 20 client.4323.objecter put_session s=0x7f566c072700 osd=0 3 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 15 client.4323.objecter _session_command_op_remove -1 1 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 20 client.4323.objecter _get_session s=0x7f566c072700 osd=0 3 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 20 client.4323.objecter get_session s=0x7f566c072700 osd=0 3 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 15 client.4323.objecter _session_command_op_assign 0 1 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 20 client.4323.objecter put_session s=0x7f566c072700 osd=0 4 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 10 client.4323.objecter _send_command 1 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 1 -- 172.21.15.110:0/3098588689 --> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f566c0758a0 con 0x7f566c072dd0 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 20 client.4323.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 1 -- 172.21.15.110:0/3098588689 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56800ce500 con 0x7f5678034d00 2024-01-27T03:43:32.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f5667fff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-27T03:43:32.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f6bfa7fc640 1 -- 172.21.15.110:0/425095038 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f6bf4017020 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.401+0000 7f6bfa7fc640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f6be806c9f0 0x7f6be806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.402+0000 7f567f7fe640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7f566c072dd0 0x7f566c0751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.402+0000 7f6c091da640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f6be806c9f0 0x7f6be806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.402+0000 7f6c091da640 10 monclient: get_auth_request con 0x7f6be806c9f0 auth_method 0 2024-01-27T03:43:32.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.402+0000 7f567f7fe640 10 monclient: get_auth_request con 0x7f566c072dd0 auth_method 0 2024-01-27T03:43:32.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.402+0000 7f6bfa7fc640 10 client.4280.objecter ms_handle_connect 0x7f6be806c9f0 2024-01-27T03:43:32.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.402+0000 7f6c091da640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f6be806c9f0 0x7f6be806eeb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6c00036000 tx=0x7f6c0003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 1 -- 172.21.15.110:0/425095038 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6bf40010c0 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 10 client.4280.objecter ms_dispatch 0x7f6c04000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 3 client.4280.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 3 client.4280.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 10 client.4280.objecter checking command 1 2024-01-27T03:43:32.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6be8072dd0 0x7f6be80751f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 20 client.4280.objecter _get_session s=0x7f6be8072700 osd=1 3 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 20 client.4280.objecter put_session s=0x7f6be8072700 osd=1 3 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 15 client.4280.objecter _session_command_op_remove -1 1 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 20 client.4280.objecter _get_session s=0x7f6be8072700 osd=1 3 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 20 client.4280.objecter get_session s=0x7f6be8072700 osd=1 3 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 15 client.4280.objecter _session_command_op_assign 1 1 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 20 client.4280.objecter put_session s=0x7f6be8072700 osd=1 4 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 10 client.4280.objecter _send_command 1 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 1 -- 172.21.15.110:0/425095038 --> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6be80758a0 con 0x7f6be8072dd0 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 20 client.4280.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 1 -- 172.21.15.110:0/425095038 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6bf4058210 con 0x7f6bfc0c32d0 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.406+0000 7f6bfa7fc640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-27T03:43:32.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.407+0000 7f6c0a1dc640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6be8072dd0 0x7f6be80751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.407+0000 7f6c0a1dc640 10 monclient: get_auth_request con 0x7f6be8072dd0 auth_method 0 2024-01-27T03:43:32.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.408+0000 7f5667fff640 10 client.4323.objecter ms_handle_connect 0x7f566c072dd0 2024-01-27T03:43:32.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.408+0000 7f567f7fe640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7f566c072dd0 0x7f566c0751f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.448 INFO:teuthology.orchestra.run.smithi110.stdout:55834574854 2024-01-27T03:43:32.449 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-27T03:43:32.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.411+0000 7f6bfa7fc640 10 client.4280.objecter ms_handle_connect 0x7f6be8072dd0 2024-01-27T03:43:32.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.411+0000 7f6c0a1dc640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6be8072dd0 0x7f6be80751f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.411+0000 7f77f6ffd640 1 -- 172.21.15.110:0/259790215 <== osd.5 v2:172.21.15.162:6802/946981202 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f77e40758a0 con 0x7f77e4072dd0 2024-01-27T03:43:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.411+0000 7f77f6ffd640 10 client.4274.objecter ms_dispatch 0x7f7800000b90 command_reply(tid 1: 0 ) v1 2024-01-27T03:43:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.411+0000 7f77f6ffd640 10 client.4274.objecter _finish_command 1 = system:0 2024-01-27T03:43:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.411+0000 7f77f6ffd640 20 client.4274.objecter put_session s=0x7f77e4072700 osd=5 4 2024-01-27T03:43:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.411+0000 7f77f6ffd640 15 client.4274.objecter _session_command_op_remove 5 1 2024-01-27T03:43:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 10 client.4277.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-27T03:43:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 15 client.4277.objecter _session_command_op_assign -1 2 2024-01-27T03:43:32.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 20 client.4277.objecter _get_session s=0x7ff3e80726e0 osd=4 3 2024-01-27T03:43:32.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 20 client.4277.objecter put_session s=0x7ff3e80726e0 osd=4 3 2024-01-27T03:43:32.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 20 client.4277.objecter _get_session s=0x7ff3e80726e0 osd=4 3 2024-01-27T03:43:32.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 15 client.4277.objecter _session_command_op_remove -1 2 2024-01-27T03:43:32.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 20 client.4277.objecter get_session s=0x7ff3e80726e0 osd=4 3 2024-01-27T03:43:32.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 15 client.4277.objecter _session_command_op_assign 4 2 2024-01-27T03:43:32.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 20 client.4277.objecter put_session s=0x7ff3e80726e0 osd=4 4 2024-01-27T03:43:32.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 10 client.4277.objecter _send_command 2 2024-01-27T03:43:32.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.416+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 --> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff4040b0f20 con 0x7ff3e8072db0 2024-01-27T03:43:32.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 10 client.4317.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 15 client.4317.objecter _session_command_op_assign -1 2 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 20 client.4317.objecter _get_session s=0x7f118c00f040 osd=3 3 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 20 client.4317.objecter put_session s=0x7f118c00f040 osd=3 3 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 20 client.4317.objecter _get_session s=0x7f118c00f040 osd=3 3 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 15 client.4317.objecter _session_command_op_remove -1 2 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 20 client.4317.objecter get_session s=0x7f118c00f040 osd=3 3 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 15 client.4317.objecter _session_command_op_assign 3 2 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 20 client.4317.objecter put_session s=0x7f118c00f040 osd=3 4 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 10 client.4317.objecter _send_command 2 2024-01-27T03:43:32.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.418+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 --> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f119413aee0 con 0x7f1178072c40 2024-01-27T03:43:32.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.427+0000 7ff3fa7fc640 1 -- 172.21.15.110:0/598992525 <== osd.4 v2:172.21.15.162:6826/1007517023 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff3fc00f040 con 0x7ff3e8072db0 2024-01-27T03:43:32.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.427+0000 7ff3fa7fc640 10 client.4277.objecter ms_dispatch 0x7ff404000b90 command_reply(tid 2: 0 ) v1 2024-01-27T03:43:32.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.427+0000 7ff3fa7fc640 10 client.4277.objecter _finish_command 2 = system:0 2024-01-27T03:43:32.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.427+0000 7ff3fa7fc640 20 client.4277.objecter put_session s=0x7ff3e80726e0 osd=4 4 2024-01-27T03:43:32.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.427+0000 7ff3fa7fc640 15 client.4277.objecter _session_command_op_remove 4 2 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 10 librados: watch_flush enter 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 10 librados: watch_flush exit 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 10 client.4277.objecter close_session for osd.4 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7ff3e8072db0 msgr2=0x7ff3e80751d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7ff3e8072db0 0x7ff3e80751d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 20 client.4277.objecter put_session s=0x7ff3e80726e0 osd=4 1 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 20 client.4277.objecter shutdown clearing up homeless session... 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 10 client.4277.objecter successfully canceled tick 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff3e806c9d0 msgr2=0x7ff3e806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff3e806c9d0 0x7ff3e806ee90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff400007fd0 tx=0x7ff400009040 comp rx=0 tx=0).stop 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 10 monclient: shutdown 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4040b0ab0 msgr2=0x7ff404150890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 1 --2- 172.21.15.110:0/598992525 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4040b0ab0 0x7ff404150890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4033000 tx=0x7ff3f40046c0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40a6ba640 1 -- 172.21.15.110:0/598992525 reap_dead start 2024-01-27T03:43:32.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 shutdown_connections 2024-01-27T03:43:32.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.428+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 >> 172.21.15.110:0/598992525 conn(0x7ff4040b1400 msgr2=0x7ff40406b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-27T03:43:32.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 15 client.4274.objecter _session_command_op_assign -1 2 2024-01-27T03:43:32.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 20 client.4274.objecter _get_session s=0x7f77e4072700 osd=5 3 2024-01-27T03:43:32.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 20 client.4274.objecter put_session s=0x7f77e4072700 osd=5 3 2024-01-27T03:43:32.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 20 client.4274.objecter _get_session s=0x7f77e4072700 osd=5 3 2024-01-27T03:43:32.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 15 client.4274.objecter _session_command_op_remove -1 2 2024-01-27T03:43:32.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 20 client.4274.objecter get_session s=0x7f77e4072700 osd=5 3 2024-01-27T03:43:32.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 15 client.4274.objecter _session_command_op_assign 5 2 2024-01-27T03:43:32.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 20 client.4274.objecter put_session s=0x7f77e4072700 osd=5 4 2024-01-27T03:43:32.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 10 client.4274.objecter _send_command 2 2024-01-27T03:43:32.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 --> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f78000b1910 con 0x7f77e4072dd0 2024-01-27T03:43:32.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 shutdown_connections 2024-01-27T03:43:32.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.429+0000 7ff40c11d640 1 -- 172.21.15.110:0/598992525 wait complete. 2024-01-27T03:43:32.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.430+0000 7ff40c11d640 1 librados: shutdown 2024-01-27T03:43:32.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.435+0000 7f77f6ffd640 1 -- 172.21.15.110:0/259790215 <== osd.5 v2:172.21.15.162:6802/946981202 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f77f0009200 con 0x7f77e4072dd0 2024-01-27T03:43:32.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.435+0000 7f77f6ffd640 10 client.4274.objecter ms_dispatch 0x7f7800000b90 command_reply(tid 2: 0 ) v1 2024-01-27T03:43:32.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.435+0000 7f77f6ffd640 10 client.4274.objecter _finish_command 2 = system:0 2024-01-27T03:43:32.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.435+0000 7f77f6ffd640 20 client.4274.objecter put_session s=0x7f77e4072700 osd=5 4 2024-01-27T03:43:32.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.435+0000 7f77f6ffd640 15 client.4274.objecter _session_command_op_remove 5 2 2024-01-27T03:43:32.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 10 librados: watch_flush enter 2024-01-27T03:43:32.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 10 librados: watch_flush exit 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 10 client.4274.objecter close_session for osd.5 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f77e4072dd0 msgr2=0x7f77e40751f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f77e4072dd0 0x7f77e40751f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 20 client.4274.objecter put_session s=0x7f77e4072700 osd=5 1 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 20 client.4274.objecter shutdown clearing up homeless session... 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 10 client.4274.objecter successfully canceled tick 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f77e406c9f0 msgr2=0x7f77e406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f77e406c9f0 0x7f77e406eeb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f77fc035000 tx=0x7f77fc02e000 comp rx=0 tx=0).stop 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 10 monclient: shutdown 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 msgr2=0x7f780013d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 1 --2- 172.21.15.110:0/259790215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78000b4930 0x7f780013d8d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f77f800c7f0 tx=0x7f77f800ccb0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f7806a94640 1 -- 172.21.15.110:0/259790215 reap_dead start 2024-01-27T03:43:32.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 shutdown_connections 2024-01-27T03:43:32.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.436+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 >> 172.21.15.110:0/259790215 conn(0x7f7800066490 msgr2=0x7f78000b8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.440+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 shutdown_connections 2024-01-27T03:43:32.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.440+0000 7f78084f7640 1 -- 172.21.15.110:0/259790215 wait complete. 2024-01-27T03:43:32.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.441+0000 7f78084f7640 1 librados: shutdown 2024-01-27T03:43:32.460 INFO:teuthology.orchestra.run.smithi110.stdout:85899345929 2024-01-27T03:43:32.461 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-27T03:43:32.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.442+0000 7f118affd640 1 -- 172.21.15.110:0/2671337598 <== osd.3 v2:172.21.15.110:6800/2374111370 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f119413aee0 con 0x7f1178072c40 2024-01-27T03:43:32.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.442+0000 7f118affd640 10 client.4317.objecter ms_dispatch 0x7f1194000b90 command_reply(tid 2: 0 ) v1 2024-01-27T03:43:32.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.442+0000 7f118affd640 10 client.4317.objecter _finish_command 2 = system:0 2024-01-27T03:43:32.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.442+0000 7f118affd640 20 client.4317.objecter put_session s=0x7f118c00f040 osd=3 4 2024-01-27T03:43:32.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.442+0000 7f118affd640 15 client.4317.objecter _session_command_op_remove 3 2 2024-01-27T03:43:32.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 10 librados: watch_flush enter 2024-01-27T03:43:32.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 10 librados: watch_flush exit 2024-01-27T03:43:32.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 10 client.4317.objecter close_session for osd.3 2024-01-27T03:43:32.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f1178072c40 msgr2=0x7f1178075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f1178072c40 0x7f1178075060 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 20 client.4317.objecter put_session s=0x7f118c00f040 osd=3 1 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 20 client.4317.objecter shutdown clearing up homeless session... 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 10 client.4317.objecter successfully canceled tick 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f117806c9d0 msgr2=0x7f117806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f117806c9d0 0x7f117806ee90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1184009690 tx=0x7f1184006bd0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 10 monclient: shutdown 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11940ae150 msgr2=0x7f11941da750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 1 --2- 172.21.15.110:0/2671337598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11940ae150 0x7f11941da750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f118c00efc0 tx=0x7f118c00c490 comp rx=0 tx=0).stop 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119aa99640 1 -- 172.21.15.110:0/2671337598 reap_dead start 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 shutdown_connections 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.443+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 >> 172.21.15.110:0/2671337598 conn(0x7f11940b3b70 msgr2=0x7f11940b2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.444+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 shutdown_connections 2024-01-27T03:43:32.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.444+0000 7f119b29a640 1 -- 172.21.15.110:0/2671337598 wait complete. 2024-01-27T03:43:32.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.444+0000 7f119b29a640 1 librados: shutdown 2024-01-27T03:43:32.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.450+0000 7f054b7fe640 1 -- 172.21.15.110:0/1693867931 <== osd.6 v2:172.21.15.162:6810/3628034141 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f05500758a0 con 0x7f0550072dd0 2024-01-27T03:43:32.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.450+0000 7f054b7fe640 10 client.4320.objecter ms_dispatch 0x7f0564000b90 command_reply(tid 1: 0 ) v1 2024-01-27T03:43:32.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.450+0000 7f054b7fe640 10 client.4320.objecter _finish_command 1 = system:0 2024-01-27T03:43:32.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.450+0000 7f054b7fe640 20 client.4320.objecter put_session s=0x7f0550072700 osd=6 4 2024-01-27T03:43:32.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.450+0000 7f054b7fe640 15 client.4320.objecter _session_command_op_remove 6 1 2024-01-27T03:43:32.466 INFO:teuthology.orchestra.run.smithi110.stdout:68719476743 2024-01-27T03:43:32.466 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-27T03:43:32.467 INFO:teuthology.orchestra.run.smithi110.stdout:64424509447 2024-01-27T03:43:32.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-27T03:43:32.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 10 client.4320.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-27T03:43:32.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 15 client.4320.objecter _session_command_op_assign -1 2 2024-01-27T03:43:32.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 20 client.4320.objecter _get_session s=0x7f0550072700 osd=6 3 2024-01-27T03:43:32.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 20 client.4320.objecter put_session s=0x7f0550072700 osd=6 3 2024-01-27T03:43:32.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 20 client.4320.objecter _get_session s=0x7f0550072700 osd=6 3 2024-01-27T03:43:32.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 15 client.4320.objecter _session_command_op_remove -1 2 2024-01-27T03:43:32.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 20 client.4320.objecter get_session s=0x7f0550072700 osd=6 3 2024-01-27T03:43:32.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 15 client.4320.objecter _session_command_op_assign 6 2 2024-01-27T03:43:32.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 20 client.4320.objecter put_session s=0x7f0550072700 osd=6 4 2024-01-27T03:43:32.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 10 client.4320.objecter _send_command 2 2024-01-27T03:43:32.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.465+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 --> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f055c004b30 con 0x7f0550072dd0 2024-01-27T03:43:32.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f054b7fe640 1 -- 172.21.15.110:0/1693867931 <== osd.6 v2:172.21.15.162:6810/3628034141 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f055c004b30 con 0x7f0550072dd0 2024-01-27T03:43:32.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f054b7fe640 10 client.4320.objecter ms_dispatch 0x7f0564000b90 command_reply(tid 2: 0 ) v1 2024-01-27T03:43:32.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f054b7fe640 10 client.4320.objecter _finish_command 2 = system:0 2024-01-27T03:43:32.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f054b7fe640 20 client.4320.objecter put_session s=0x7f0550072700 osd=6 4 2024-01-27T03:43:32.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f054b7fe640 15 client.4320.objecter _session_command_op_remove 6 2 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 10 librados: watch_flush enter 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 10 librados: watch_flush exit 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 10 client.4320.objecter close_session for osd.6 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f0550072dd0 msgr2=0x7f05500751f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f0550072dd0 0x7f05500751f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 20 client.4320.objecter put_session s=0x7f0550072700 osd=6 1 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 20 client.4320.objecter shutdown clearing up homeless session... 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 10 client.4320.objecter successfully canceled tick 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f055006c9f0 msgr2=0x7f055006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f055006c9f0 0x7f055006eeb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0554000f60 tx=0x7f0554009040 comp rx=0 tx=0).stop 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 10 monclient: shutdown 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f055c0c6da0 msgr2=0x7f055c0c0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f0563fff640 1 --2- 172.21.15.110:0/1693867931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f055c0c6da0 0x7f055c0c0d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0564073520 tx=0x7f0564068370 comp rx=0 tx=0).stop 2024-01-27T03:43:32.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.476+0000 7f05637fe640 1 -- 172.21.15.110:0/1693867931 reap_dead start 2024-01-27T03:43:32.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.477+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 shutdown_connections 2024-01-27T03:43:32.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.477+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 >> 172.21.15.110:0/1693867931 conn(0x7f055c0061a0 msgr2=0x7f055c0c1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.477+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 shutdown_connections 2024-01-27T03:43:32.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.477+0000 7f0563fff640 1 -- 172.21.15.110:0/1693867931 wait complete. 2024-01-27T03:43:32.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.477+0000 7f0563fff640 1 librados: shutdown 2024-01-27T03:43:32.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.485+0000 7f5667fff640 1 -- 172.21.15.110:0/3098588689 <== osd.0 v2:172.21.15.110:6816/1545415171 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f566c0758a0 con 0x7f566c072dd0 2024-01-27T03:43:32.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.485+0000 7f5667fff640 10 client.4323.objecter ms_dispatch 0x7f5680000b90 command_reply(tid 1: 0 ) v1 2024-01-27T03:43:32.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.485+0000 7f5667fff640 10 client.4323.objecter _finish_command 1 = system:0 2024-01-27T03:43:32.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.485+0000 7f5667fff640 20 client.4323.objecter put_session s=0x7f566c072700 osd=0 4 2024-01-27T03:43:32.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.485+0000 7f5667fff640 15 client.4323.objecter _session_command_op_remove 0 1 2024-01-27T03:43:32.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.486+0000 7f6bfa7fc640 1 -- 172.21.15.110:0/425095038 <== osd.1 v2:172.21.15.110:6823/518896551 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6be80758a0 con 0x7f6be8072dd0 2024-01-27T03:43:32.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.486+0000 7f6bfa7fc640 10 client.4280.objecter ms_dispatch 0x7f6c04000b90 command_reply(tid 1: 0 ) v1 2024-01-27T03:43:32.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.486+0000 7f6bfa7fc640 10 client.4280.objecter _finish_command 1 = system:0 2024-01-27T03:43:32.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.486+0000 7f6bfa7fc640 20 client.4280.objecter put_session s=0x7f6be8072700 osd=1 4 2024-01-27T03:43:32.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.486+0000 7f6bfa7fc640 15 client.4280.objecter _session_command_op_remove 1 1 2024-01-27T03:43:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.492+0000 7fd225ffb640 1 -- 172.21.15.110:0/917050832 <== osd.2 v2:172.21.15.110:6808/3683566991 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd2140758d0 con 0x7fd214072e00 2024-01-27T03:43:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.492+0000 7fd225ffb640 10 client.4283.objecter ms_dispatch 0x7fd230000b90 command_reply(tid 1: 0 ) v1 2024-01-27T03:43:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.492+0000 7fd225ffb640 10 client.4283.objecter _finish_command 1 = system:0 2024-01-27T03:43:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.492+0000 7fd225ffb640 20 client.4283.objecter put_session s=0x7fd214072730 osd=2 4 2024-01-27T03:43:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.492+0000 7fd225ffb640 15 client.4283.objecter _session_command_op_remove 2 1 2024-01-27T03:43:32.497 INFO:teuthology.orchestra.run.smithi110.stdout:85899345931 2024-01-27T03:43:32.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 15 client.4280.objecter _session_command_op_assign -1 2 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 20 client.4280.objecter _get_session s=0x7f6be8072700 osd=1 3 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 20 client.4280.objecter put_session s=0x7f6be8072700 osd=1 3 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 20 client.4280.objecter _get_session s=0x7f6be8072700 osd=1 3 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 15 client.4280.objecter _session_command_op_remove -1 2 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 20 client.4280.objecter get_session s=0x7f6be8072700 osd=1 3 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 15 client.4280.objecter _session_command_op_assign 1 2 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 20 client.4280.objecter put_session s=0x7f6be8072700 osd=1 4 2024-01-27T03:43:32.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 10 client.4280.objecter _send_command 2 2024-01-27T03:43:32.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.510+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 --> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6bfc005c00 con 0x7f6be8072dd0 2024-01-27T03:43:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 10 client.4323.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-27T03:43:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 15 client.4323.objecter _session_command_op_assign -1 2 2024-01-27T03:43:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 20 client.4323.objecter _get_session s=0x7f566c072700 osd=0 3 2024-01-27T03:43:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 20 client.4323.objecter put_session s=0x7f566c072700 osd=0 3 2024-01-27T03:43:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 20 client.4323.objecter _get_session s=0x7f566c072700 osd=0 3 2024-01-27T03:43:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 15 client.4323.objecter _session_command_op_remove -1 2 2024-01-27T03:43:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 20 client.4323.objecter get_session s=0x7f566c072700 osd=0 3 2024-01-27T03:43:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 15 client.4323.objecter _session_command_op_assign 0 2 2024-01-27T03:43:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 20 client.4323.objecter put_session s=0x7f566c072700 osd=0 4 2024-01-27T03:43:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 10 client.4323.objecter _send_command 2 2024-01-27T03:43:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.515+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 --> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f56780307a0 con 0x7f566c072dd0 2024-01-27T03:43:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 10 client.4283.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-27T03:43:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 15 client.4283.objecter _session_command_op_assign -1 2 2024-01-27T03:43:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 20 client.4283.objecter _get_session s=0x7fd214072730 osd=2 3 2024-01-27T03:43:32.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 20 client.4283.objecter put_session s=0x7fd214072730 osd=2 3 2024-01-27T03:43:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 20 client.4283.objecter _get_session s=0x7fd214072730 osd=2 3 2024-01-27T03:43:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 15 client.4283.objecter _session_command_op_remove -1 2 2024-01-27T03:43:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 20 client.4283.objecter get_session s=0x7fd214072730 osd=2 3 2024-01-27T03:43:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 15 client.4283.objecter _session_command_op_assign 2 2 2024-01-27T03:43:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 20 client.4283.objecter put_session s=0x7fd214072730 osd=2 4 2024-01-27T03:43:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 10 client.4283.objecter _send_command 2 2024-01-27T03:43:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.521+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 --> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd2300ad680 con 0x7fd214072e00 2024-01-27T03:43:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.528+0000 7f6bfa7fc640 1 -- 172.21.15.110:0/425095038 <== osd.1 v2:172.21.15.110:6823/518896551 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6bfc005c00 con 0x7f6be8072dd0 2024-01-27T03:43:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.528+0000 7f6bfa7fc640 10 client.4280.objecter ms_dispatch 0x7f6c04000b90 command_reply(tid 2: 0 ) v1 2024-01-27T03:43:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.528+0000 7f6bfa7fc640 10 client.4280.objecter _finish_command 2 = system:0 2024-01-27T03:43:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.528+0000 7f6bfa7fc640 20 client.4280.objecter put_session s=0x7f6be8072700 osd=1 4 2024-01-27T03:43:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.528+0000 7f6bfa7fc640 15 client.4280.objecter _session_command_op_remove 1 2 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.529+0000 7f6c0bc3f640 10 librados: watch_flush enter 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.529+0000 7f6c0bc3f640 10 librados: watch_flush exit 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.529+0000 7f6c0bc3f640 10 client.4280.objecter close_session for osd.1 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.529+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6be8072dd0 msgr2=0x7f6be80751f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.529+0000 7f6c0bc3f640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6be8072dd0 0x7f6be80751f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.529+0000 7f6c0bc3f640 20 client.4280.objecter put_session s=0x7f6be8072700 osd=1 1 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.529+0000 7f6c0bc3f640 20 client.4280.objecter shutdown clearing up homeless session... 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.529+0000 7f6c0bc3f640 10 client.4280.objecter successfully canceled tick 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f6be806c9f0 msgr2=0x7f6be806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f6be806c9f0 0x7f6be806eeb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6c00036000 tx=0x7f6c0003d040 comp rx=0 tx=0).stop 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 10 monclient: shutdown 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bfc0c32d0 msgr2=0x7f6bfc0c7e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 1 --2- 172.21.15.110:0/425095038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bfc0c32d0 0x7f6bfc0c7e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400c410 tx=0x7f6bf400c8d0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0a1dc640 1 -- 172.21.15.110:0/425095038 reap_dead start 2024-01-27T03:43:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 shutdown_connections 2024-01-27T03:43:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f5667fff640 1 -- 172.21.15.110:0/3098588689 <== osd.0 v2:172.21.15.110:6816/1545415171 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5670031b00 con 0x7f566c072dd0 2024-01-27T03:43:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f5667fff640 10 client.4323.objecter ms_dispatch 0x7f5680000b90 command_reply(tid 2: 0 ) v1 2024-01-27T03:43:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f5667fff640 10 client.4323.objecter _finish_command 2 = system:0 2024-01-27T03:43:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 >> 172.21.15.110:0/425095038 conn(0x7f6bfc0061e0 msgr2=0x7f6bfc0c1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f5667fff640 20 client.4323.objecter put_session s=0x7f566c072700 osd=0 4 2024-01-27T03:43:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f5667fff640 15 client.4323.objecter _session_command_op_remove 0 2 2024-01-27T03:43:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 shutdown_connections 2024-01-27T03:43:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 1 -- 172.21.15.110:0/425095038 wait complete. 2024-01-27T03:43:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.531+0000 7f6c0bc3f640 1 librados: shutdown 2024-01-27T03:43:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.532+0000 7f567ffff640 10 librados: watch_flush enter 2024-01-27T03:43:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.532+0000 7f567ffff640 10 librados: watch_flush exit 2024-01-27T03:43:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.532+0000 7f567ffff640 10 client.4323.objecter close_session for osd.0 2024-01-27T03:43:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.532+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7f566c072dd0 msgr2=0x7f566c0751f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.532+0000 7f567ffff640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7f566c072dd0 0x7f566c0751f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.532+0000 7f567ffff640 20 client.4323.objecter put_session s=0x7f566c072700 osd=0 1 2024-01-27T03:43:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.532+0000 7f567ffff640 20 client.4323.objecter shutdown clearing up homeless session... 2024-01-27T03:43:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.532+0000 7f567ffff640 10 client.4323.objecter successfully canceled tick 2024-01-27T03:43:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f566c06c9f0 msgr2=0x7f566c06eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f566c06c9f0 0x7f566c06eeb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f56780303c0 tx=0x7f5674007930 comp rx=0 tx=0).stop 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 10 monclient: shutdown 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5678034d00 msgr2=0x7f5678155500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 1 --2- 172.21.15.110:0/3098588689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5678034d00 0x7f5678155500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f568004f800 tx=0x7f5680068370 comp rx=0 tx=0).stop 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567f7fe640 1 -- 172.21.15.110:0/3098588689 reap_dead start 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 shutdown_connections 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 >> 172.21.15.110:0/3098588689 conn(0x7f56780061a0 msgr2=0x7f5678006580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 shutdown_connections 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.536+0000 7f567ffff640 1 -- 172.21.15.110:0/3098588689 wait complete. 2024-01-27T03:43:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.537+0000 7f567ffff640 1 librados: shutdown 2024-01-27T03:43:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd225ffb640 1 -- 172.21.15.110:0/917050832 <== osd.2 v2:172.21.15.110:6808/3683566991 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd2300ad680 con 0x7fd214072e00 2024-01-27T03:43:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd225ffb640 10 client.4283.objecter ms_dispatch 0x7fd230000b90 command_reply(tid 2: 0 ) v1 2024-01-27T03:43:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd225ffb640 10 client.4283.objecter _finish_command 2 = system:0 2024-01-27T03:43:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd225ffb640 20 client.4283.objecter put_session s=0x7fd214072730 osd=2 4 2024-01-27T03:43:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd225ffb640 15 client.4283.objecter _session_command_op_remove 2 2 2024-01-27T03:43:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd23733e640 10 librados: watch_flush enter 2024-01-27T03:43:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd23733e640 10 librados: watch_flush exit 2024-01-27T03:43:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd23733e640 10 client.4283.objecter close_session for osd.2 2024-01-27T03:43:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7fd214072e00 msgr2=0x7fd214075220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd23733e640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7fd214072e00 0x7fd214075220 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd23733e640 20 client.4283.objecter put_session s=0x7fd214072730 osd=2 1 2024-01-27T03:43:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd23733e640 20 client.4283.objecter shutdown clearing up homeless session... 2024-01-27T03:43:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.547+0000 7fd23733e640 10 client.4283.objecter successfully canceled tick 2024-01-27T03:43:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.549+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd21406ca20 msgr2=0x7fd21406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.549+0000 7fd23733e640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fd21406ca20 0x7fd21406eee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd22c036000 tx=0x7fd22c03d040 comp rx=0 tx=0).stop 2024-01-27T03:43:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.549+0000 7fd23733e640 10 monclient: shutdown 2024-01-27T03:43:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.549+0000 7fd23733e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.549+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300ad960 msgr2=0x7fd23013b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.549+0000 7fd23733e640 1 --2- 172.21.15.110:0/917050832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2300ad960 0x7fd23013b220 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd22800b9e0 tx=0x7fd22800bea0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.549+0000 7fd2358db640 1 -- 172.21.15.110:0/917050832 reap_dead start 2024-01-27T03:43:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.549+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 shutdown_connections 2024-01-27T03:43:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.549+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 >> 172.21.15.110:0/917050832 conn(0x7fd230066490 msgr2=0x7fd2300b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.553 INFO:teuthology.orchestra.run.smithi110.stdout:90194313225 2024-01-27T03:43:32.553 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-27T03:43:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.550+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 shutdown_connections 2024-01-27T03:43:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.552+0000 7fd23733e640 1 -- 172.21.15.110:0/917050832 wait complete. 2024-01-27T03:43:32.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.552+0000 7fd23733e640 1 librados: shutdown 2024-01-27T03:43:32.565 INFO:teuthology.orchestra.run.smithi110.stdout:68719476743 2024-01-27T03:43:32.565 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-27T03:43:32.572 INFO:teuthology.orchestra.run.smithi110.stdout:90194313225 2024-01-27T03:43:32.572 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-27T03:43:32.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.608+0000 7fe61ff47640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.608+0000 7fe61ff47640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.608+0000 7fe61ff47640 10 monclient: monmap: 2024-01-27T03:43:32.609 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.609676+0000 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.609676+0000 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.610 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.609+0000 7fe61ff47640 1 Processor -- start 2024-01-27T03:43:32.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.609+0000 7fe61ff47640 1 -- start start 2024-01-27T03:43:32.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.609+0000 7fe61ff47640 10 monclient: init 2024-01-27T03:43:32.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.610+0000 7fe61ff47640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.610+0000 7fe61ff47640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-01-27T03:43:32.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.611+0000 7fe61ff47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6180b4680 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:32.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.611+0000 7fe61ff47640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4a60 0x7fe6180b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.612+0000 7fe61dce3640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe6180b4680 0x7fe618142970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55438/0 (socket says 172.21.15.110:55438) 2024-01-27T03:43:32.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.612+0000 7fe61dce3640 1 -- 172.21.15.110:0/2848476617 learned_addr learned my addr 172.21.15.110:0/2848476617 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:32.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.613+0000 7fe61ff47640 10 monclient(hunting): picked mon.noname-b con 0x7fe6180b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.613+0000 7fe61ff47640 1 --2- 172.21.15.110:0/2848476617 >> v2:172.21.15.110:3300/0 conn(0x7fe6180b3b30 0x7fe6180b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61d4e2640 1 --2- 172.21.15.110:0/2848476617 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4a60 0x7fe6180b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61ff47640 10 monclient(hunting): picked mon.noname-a con 0x7fe6180b3b30 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61ff47640 1 -- 172.21.15.110:0/2848476617 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180afbe0 con 0x7fe6180b4680 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61ff47640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61ff47640 1 -- 172.21.15.110:0/2848476617 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe618142240 con 0x7fe6180b3b30 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61ff47640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61ff47640 1 -- 172.21.15.110:0/2848476617 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180b0890 con 0x7fe6180b4a60 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61ff47640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61ff47640 10 monclient(hunting): authenticate will time out at 1495.968599s 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61d4e2640 10 monclient(hunting): get_auth_request con 0x7fe6180b4a60 auth_method 0 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61d4e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61d4e2640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.614+0000 7fe61d4e2640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.615+0000 7fe61e4e4640 1 --2- 172.21.15.110:0/2848476617 >> v2:172.21.15.110:3300/0 conn(0x7fe6180b3b30 0x7fe6180b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.615+0000 7fe61e4e4640 10 monclient(hunting): get_auth_request con 0x7fe6180b3b30 auth_method 0 2024-01-27T03:43:32.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.615+0000 7fe61e4e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.615+0000 7fe61e4e4640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.615+0000 7fe61e4e4640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.616+0000 7fe61d4e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.616+0000 7fe61d4e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.616+0000 7fe61d4e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.619+0000 7fe61e4e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.619+0000 7fe61e4e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.619+0000 7fe61e4e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.619+0000 7fe61d4e2640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2024-01-27T03:43:32.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.619+0000 7fe61d4e2640 1 -- 172.21.15.110:0/2848476617 >> v2:172.21.15.110:3300/0 conn(0x7fe6180b3b30 msgr2=0x7fe6180b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.620+0000 7fe61cce1640 1 -- 172.21.15.110:0/2848476617 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036878156 0 0) 0x7fe6180afbe0 con 0x7fe6180b4680 2024-01-27T03:43:32.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.620+0000 7fe61d4e2640 1 --2- 172.21.15.110:0/2848476617 >> v2:172.21.15.110:3300/0 conn(0x7fe6180b3b30 0x7fe6180b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.620+0000 7fe61d4e2640 1 -- 172.21.15.110:0/2848476617 >> v1:172.21.15.110:6789/0 conn(0x7fe6180b4680 legacy=0x7fe618142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.620+0000 7fe61d4e2640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.620+0000 7fe61d4e2640 1 monclient: found mon.noname-b 2024-01-27T03:43:32.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.620+0000 7fe61d4e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.620+0000 7fe61d4e2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.620+0000 7fe61d4e2640 1 -- 172.21.15.110:0/2848476617 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6180b0b00 con 0x7fe6180b4a60 2024-01-27T03:43:32.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.622+0000 7fe61cce1640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-27T03:43:32.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.625+0000 7fe61d4e2640 1 --2- 172.21.15.110:0/2848476617 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4a60 0x7fe6180b1470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe6080097a0 tx=0x7fe60802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff07f66ccb6ada3e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.626+0000 7fae5132e640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.626+0000 7fae5132e640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.626+0000 7fae5132e640 10 monclient: monmap: 2024-01-27T03:43:32.628 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.628 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.628 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.627412+0000 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.627412+0000 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.628+0000 7fae5132e640 1 Processor -- start 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.628+0000 7fae5132e640 1 -- start start 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.628+0000 7fae5132e640 10 monclient: init 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae4c0aed60 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fae4c072a50 0x7fae4c0acef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 10 monclient(hunting): picked mon.noname-c con 0x7fae4c072a50 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fae4c0ada30 0x7fae4c13ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 10 monclient(hunting): picked mon.noname-a con 0x7fae4c0ada30 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4c0adec0 con 0x7fae4c0aed60 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0af160 con 0x7fae4c0ada30 2024-01-27T03:43:32.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0ab240 con 0x7fae4c072a50 2024-01-27T03:43:32.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.629+0000 7fae5132e640 10 monclient(hunting): authenticate will time out at 1495.983747s 2024-01-27T03:43:32.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.630+0000 7fe61cce1640 1 -- 172.21.15.110:0/2848476617 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6080317c0 con 0x7fe6180b4a60 2024-01-27T03:43:32.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:32.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 10 monclient: dump: 2024-01-27T03:43:32.633 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 1 -- 172.21.15.110:0/2848476617 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe608031920 con 0x7fe6180b4a60 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 1 -- 172.21.15.110:0/2848476617 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608031cf0 con 0x7fe6180b4a60 2024-01-27T03:43:32.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61cce1640 10 monclient: dump: 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61ff47640 5 monclient: authenticate success, global_id 4286 2024-01-27T03:43:32.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61ff47640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fe61ff47640 4 set_mon_vals no callback set 2024-01-27T03:43:32.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fe61ff47640 10 monclient: shutdown 2024-01-27T03:43:32.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fe61ff47640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fe61ff47640 1 -- 172.21.15.110:0/2848476617 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4a60 msgr2=0x7fe6180b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fe61ff47640 1 --2- 172.21.15.110:0/2848476617 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4a60 0x7fe6180b1470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe6080097a0 tx=0x7fe60802b860 comp rx=0 tx=0).stop 2024-01-27T03:43:32.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fae4b7fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fae4c072a50 0x7fae4c0acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fae4b7fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fae4c072a50 0x7fae4c0acef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40704/0 (socket says 172.21.15.110:40704) 2024-01-27T03:43:32.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fae4b7fe640 1 -- 172.21.15.110:0/682678747 learned_addr learned my addr 172.21.15.110:0/682678747 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fae4b7fe640 10 monclient(hunting): get_auth_request con 0x7fae4c072a50 auth_method 0 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fae4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fae4b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.631+0000 7fae4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fae4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fae4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fae4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fae50b2d640 1 --2- 172.21.15.110:0/682678747 >> v2:172.21.15.110:3300/0 conn(0x7fae4c0ada30 0x7fae4c13ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fae50b2d640 10 monclient(hunting): get_auth_request con 0x7fae4c0ada30 auth_method 0 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fae50b2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fae50b2d640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.632+0000 7fae50b2d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fe61ff47640 1 -- 172.21.15.110:0/2848476617 shutdown_connections 2024-01-27T03:43:32.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fe61ff47640 1 --2- 172.21.15.110:0/2848476617 >> v2:172.21.15.110:3300/0 conn(0x7fe6180b3b30 0x7fe6180b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fe61ff47640 1 --2- 172.21.15.110:0/2848476617 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4a60 0x7fe6180b1470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fe61ff47640 1 -- 172.21.15.110:0/2848476617 >> 172.21.15.110:0/2848476617 conn(0x7fe618066490 msgr2=0x7fe6180668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fe61ff47640 1 -- 172.21.15.110:0/2848476617 shutdown_connections 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fe61ff47640 1 -- 172.21.15.110:0/2848476617 wait complete. 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fae4b7fe640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fae4b7fe640 1 -- 172.21.15.110:0/682678747 >> v2:172.21.15.110:3300/0 conn(0x7fae4c0ada30 msgr2=0x7fae4c13ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fae4b7fe640 1 --2- 172.21.15.110:0/682678747 >> v2:172.21.15.110:3300/0 conn(0x7fae4c0ada30 0x7fae4c13ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fae4b7fe640 1 -- 172.21.15.110:0/682678747 >> v1:172.21.15.110:6789/0 conn(0x7fae4c0aed60 legacy=0x7fae4c0aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fae4b7fe640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fae4b7fe640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fae4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fae4b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.635+0000 7fae4b7fe640 1 -- 172.21.15.110:0/682678747 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c0ac7f0 con 0x7fae4c072a50 2024-01-27T03:43:32.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.637+0000 7fe61ff47640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.637+0000 7fe61ff47640 10 monclient: monmap: 2024-01-27T03:43:32.641 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.641 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.641 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.641 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.641 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.641 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.641 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.637+0000 7fe61ff47640 1 librados: starting msgr at 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.637+0000 7fe61ff47640 1 librados: starting objecter 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.638+0000 7fe61ff47640 1 Processor -- start 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.638+0000 7fe61ff47640 1 -- start start 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.638+0000 7fe61ff47640 1 librados: setting wanted keys 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.638+0000 7fe61ff47640 1 librados: calling monclient init 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.638+0000 7fe61ff47640 10 monclient: init 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.639+0000 7fe61ff47640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.639+0000 7fe61ff47640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.639+0000 7fe61ff47640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6180b3b30 0x7fe61813a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.639+0000 7fe61ff47640 10 monclient(hunting): picked mon.noname-a con 0x7fe6180b3b30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.639+0000 7fe61ff47640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4680 0x7fe61813ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.640+0000 7fae4b7fe640 1 --2- 172.21.15.110:0/682678747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c072a50 0x7fae4c0acef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0b4750 tx=0x7fae44030570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2ccccab7bf2b743 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.640+0000 7fe61ff47640 10 monclient(hunting): picked mon.noname-c con 0x7fe6180b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.640+0000 7fe61ff47640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61813d870 0x7fe6181d71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.640+0000 7fe61ff47640 10 monclient(hunting): picked mon.noname-b con 0x7fe61813d870 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.640+0000 7fe61ff47640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.640+0000 7fe61ff47640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61813e970 con 0x7fe6180b3b30 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.640+0000 7fe61ff47640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.640+0000 7fe61ff47640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61813e450 con 0x7fe61813d870 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.641+0000 7fe61ff47640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.641+0000 7fe61ff47640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181d76f0 con 0x7fe6180b4680 2024-01-27T03:43:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.641+0000 7fe61ff47640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.641+0000 7fe61ff47640 10 monclient(hunting): authenticate will time out at 1495.995072s 2024-01-27T03:43:32.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.641+0000 7f11224d7640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7f11224d7640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7f11224d7640 10 monclient: monmap: 2024-01-27T03:43:32.645 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.645 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.645 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.643189+0000 2024-01-27T03:43:32.646 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.643189+0000 2024-01-27T03:43:32.646 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.646 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.646 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.646 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.646 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.646 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.646 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.646 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7fe61d4e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4680 0x7fe61813ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7fe61dce3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6180b3b30 0x7fe61813a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7fe61d4e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4680 0x7fe61813ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38424/0 (socket says 172.21.15.110:38424) 2024-01-27T03:43:32.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7fe61dce3640 10 monclient(hunting): get_auth_request con 0x7fe6180b3b30 auth_method 0 2024-01-27T03:43:32.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7fe61dce3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7fe61dce3640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7fe61dce3640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.643+0000 7f11224d7640 1 Processor -- start 2024-01-27T03:43:32.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.643+0000 7f11224d7640 1 -- start start 2024-01-27T03:43:32.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.643+0000 7f11224d7640 10 monclient: init 2024-01-27T03:43:32.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae4affd640 1 -- 172.21.15.110:0/682678747 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44039410 con 0x7fae4c072a50 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae4affd640 10 monclient: dump: 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.650 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.651 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.651 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae4affd640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae4affd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae5132e640 5 monclient: authenticate success, global_id 4324 2024-01-27T03:43:32.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-27T03:43:32.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f111c0b4680 0x7f111c142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 10 monclient(hunting): picked mon.noname-a con 0x7f111c0b4680 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:32.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.642+0000 7fe61d4e2640 1 -- 172.21.15.110:0/3525405195 learned_addr learned my addr 172.21.15.110:0/3525405195 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fe61d4e2640 10 monclient(hunting): get_auth_request con 0x7fe6180b4680 auth_method 0 2024-01-27T03:43:32.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fe61d4e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fe61d4e2640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fe61d4e2640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fe61dce3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fe61dce3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fe61dce3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.645+0000 7fe61e4e4640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61813d870 0x7fe6181d71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.645+0000 7fe61d4e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.645+0000 7fe61d4e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.645+0000 7fe61d4e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fe61e4e4640 10 monclient(hunting): get_auth_request con 0x7fe61813d870 auth_method 0 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fe61e4e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fe61e4e4640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fe61e4e4640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae5132e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae4affd640 1 -- 172.21.15.110:0/682678747 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae44033030 con 0x7fae4c072a50 2024-01-27T03:43:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7fae4affd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.645+0000 7fae5132e640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.645+0000 7fae5132e640 4 set_mon_vals no callback set 2024-01-27T03:43:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fae5132e640 10 monclient: shutdown 2024-01-27T03:43:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fae5132e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fae5132e640 1 -- 172.21.15.110:0/682678747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c072a50 msgr2=0x7fae4c0acef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fae5132e640 1 --2- 172.21.15.110:0/682678747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c072a50 0x7fae4c0acef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0b4750 tx=0x7fae44030570 comp rx=0 tx=0).stop 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f111c0b4a60 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 0x7f111c0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 10 monclient(hunting): picked mon.noname-b con 0x7f111c142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c0afbe0 con 0x7f111c0b4a60 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111c142240 con 0x7f111c0b4680 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c0b0890 con 0x7f111c142eb0 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.644+0000 7f11224d7640 10 monclient(hunting): authenticate will time out at 1495.998210s 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7f1120a74640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 0x7f111c0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7f111b7fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f111c0b4a60 0x7f111c0b1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:49146/0 (socket says 172.21.15.110:49146) 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7f1120a74640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 0x7f111c0b1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38430/0 (socket says 172.21.15.110:38430) 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7f111b7fe640 1 -- 172.21.15.110:0/1633267756 learned_addr learned my addr 172.21.15.110:0/1633267756 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7f1120a74640 10 monclient(hunting): get_auth_request con 0x7f111c142eb0 auth_method 0 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7f1120a74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7f1120a74640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7f1120a74640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fe61e4e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fe61e4e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.646+0000 7fe61e4e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.647+0000 7fae5132e640 1 -- 172.21.15.110:0/682678747 shutdown_connections 2024-01-27T03:43:32.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.647+0000 7fae5132e640 1 --2- 172.21.15.110:0/682678747 >> v2:172.21.15.110:3300/0 conn(0x7fae4c0ada30 0x7fae4c13ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.647+0000 7fae5132e640 1 --2- 172.21.15.110:0/682678747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c072a50 0x7fae4c0acef0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.647+0000 7fae5132e640 1 -- 172.21.15.110:0/682678747 >> 172.21.15.110:0/682678747 conn(0x7fae4c0b3b70 msgr2=0x7fae4c0b3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.647+0000 7fae5132e640 1 -- 172.21.15.110:0/682678747 shutdown_connections 2024-01-27T03:43:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.647+0000 7f85d53d2640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.647+0000 7f85d53d2640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.647+0000 7f85d53d2640 10 monclient: monmap: 2024-01-27T03:43:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.648924+0000 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.648924+0000 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.661 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.648+0000 7fae5132e640 1 -- 172.21.15.110:0/682678747 wait complete. 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.649+0000 7fae5132e640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.649+0000 7fae5132e640 10 monclient: monmap: 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.662 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.649+0000 7fae5132e640 1 librados: starting msgr at 2024-01-27T03:43:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.649+0000 7fae5132e640 1 librados: starting objecter 2024-01-27T03:43:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.649+0000 7fae5132e640 1 Processor -- start 2024-01-27T03:43:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.650+0000 7f85d53d2640 1 Processor -- start 2024-01-27T03:43:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.650+0000 7f85d53d2640 1 -- start start 2024-01-27T03:43:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.650+0000 7f85d53d2640 10 monclient: init 2024-01-27T03:43:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.650+0000 7fae5132e640 1 -- start start 2024-01-27T03:43:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.650+0000 7fae5132e640 1 librados: setting wanted keys 2024-01-27T03:43:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.650+0000 7fae5132e640 1 librados: calling monclient init 2024-01-27T03:43:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.650+0000 7fae5132e640 10 monclient: init 2024-01-27T03:43:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:43:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c072a50 0x7fae4c1c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient(hunting): picked mon.noname-a con 0x7fae4c072a50 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-01-27T03:43:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85d00a98b0 0x7f85d0143740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient(hunting): picked mon.noname-b con 0x7f85d00a98b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f85d00a9c90 0x7f85d0143c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient(hunting): picked mon.noname-a con 0x7f85d00a9c90 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f85d00aaaf0 0x7f85d01468d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient(hunting): picked mon.noname-c con 0x7f85d00aaaf0 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d00abbc0 con 0x7f85d00a9c90 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d00aa6a0 con 0x7f85d00aaaf0 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00ac250 con 0x7f85d00a98b0 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7f85d53d2640 10 monclient(hunting): authenticate will time out at 1496.005717s 2024-01-27T03:43:32.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c0ada30 0x7fae4c13bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient(hunting): picked mon.noname-b con 0x7fae4c0ada30 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae4c0aed60 0x7fae4c13c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient(hunting): picked mon.noname-c con 0x7fae4c0aed60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c1c77a0 con 0x7fae4c072a50 2024-01-27T03:43:32.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c1c7a10 con 0x7fae4c0ada30 2024-01-27T03:43:32.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c13ded0 con 0x7fae4c0aed60 2024-01-27T03:43:32.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.651+0000 7fae5132e640 10 monclient(hunting): authenticate will time out at 1496.005722s 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 1 -- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61813d870 msgr2=0x7fe6181d71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61813d870 0x7fe6181d71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 1 -- 172.21.15.110:0/3525405195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6180b3b30 msgr2=0x7fe61813a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6180b3b30 0x7fe61813a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61d4e2640 1 -- 172.21.15.110:0/3525405195 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608009400 con 0x7fe6180b4680 2024-01-27T03:43:32.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7fe61e4e4640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61813d870 0x7fe6181d71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T03:43:32.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7f1120a74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7f1120a74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.652+0000 7f1120a74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.653+0000 7f111bfff640 1 --2- 172.21.15.110:0/1633267756 >> v2:172.21.15.110:3300/0 conn(0x7f111c0b4680 0x7f111c142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.653+0000 7fae4b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c0ada30 0x7fae4c13bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.653+0000 7f111bfff640 10 monclient(hunting): get_auth_request con 0x7f111c0b4680 auth_method 0 2024-01-27T03:43:32.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.653+0000 7f111bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.653+0000 7f111bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.653+0000 7f111bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.653+0000 7f85d4bd1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f85d00aaaf0 0x7f85d01468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7fae4b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c0ada30 0x7fae4c13bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40724/0 (socket says 172.21.15.110:40724) 2024-01-27T03:43:32.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7fae4b7fe640 1 -- 172.21.15.110:0/2777553887 learned_addr learned my addr 172.21.15.110:0/2777553887 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7fae4b7fe640 10 monclient(hunting): get_auth_request con 0x7fae4c0ada30 auth_method 0 2024-01-27T03:43:32.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7fae4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7fae4b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7fae4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f85d4bd1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f85d00aaaf0 0x7f85d01468d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40726/0 (socket says 172.21.15.110:40726) 2024-01-27T03:43:32.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f85d4bd1640 1 -- 172.21.15.110:0/4269053715 learned_addr learned my addr 172.21.15.110:0/4269053715 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f85d4bd1640 10 monclient(hunting): get_auth_request con 0x7f85d00aaaf0 auth_method 0 2024-01-27T03:43:32.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f85d4bd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f85d4bd1640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f85d4bd1640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f111bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f111bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f111bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f85d4bd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.654+0000 7f85d4bd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.655+0000 7f85d4bd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.655+0000 7fae4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.655+0000 7fae4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.655+0000 7fae4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.655+0000 7fae4bfff640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c072a50 0x7fae4c1c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7fae4b7fe640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-01-27T03:43:32.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7fae4b7fe640 1 -- 172.21.15.110:0/2777553887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae4c0aed60 msgr2=0x7fae4c13c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7fae4b7fe640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae4c0aed60 0x7fae4c13c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7fae4b7fe640 1 -- 172.21.15.110:0/2777553887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c072a50 msgr2=0x7fae4c1c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7fae4b7fe640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c072a50 0x7fae4c1c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7fae4b7fe640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7fae4b7fe640 1 monclient: found mon.noname-b 2024-01-27T03:43:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7fae4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7fae4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7fae4b7fe640 1 -- 172.21.15.110:0/2777553887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae44007790 con 0x7fae4c0ada30 2024-01-27T03:43:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7f1120a74640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-01-27T03:43:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7f1120a74640 1 -- 172.21.15.110:0/1633267756 >> v2:172.21.15.110:3300/0 conn(0x7f111c0b4680 msgr2=0x7f111c142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7f1120a74640 1 --2- 172.21.15.110:0/1633267756 >> v2:172.21.15.110:3300/0 conn(0x7f111c0b4680 0x7f111c142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7f1120a74640 1 -- 172.21.15.110:0/1633267756 >> v1:172.21.15.162:6789/0 conn(0x7f111c0b4a60 legacy=0x7f111c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7f1120a74640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7f1120a74640 1 monclient: found mon.noname-b 2024-01-27T03:43:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7f1120a74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7f1120a74640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.658+0000 7f1120a74640 1 -- 172.21.15.110:0/1633267756 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c0b0b00 con 0x7f111c142eb0 2024-01-27T03:43:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2024-01-27T03:43:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 1 -- 172.21.15.110:0/4269053715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85d00a98b0 msgr2=0x7f85d0143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 1 --2- 172.21.15.110:0/4269053715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85d00a98b0 0x7f85d0143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 1 -- 172.21.15.110:0/4269053715 >> v2:172.21.15.110:3300/0 conn(0x7f85d00a9c90 msgr2=0x7f85d0143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 1 --2- 172.21.15.110:0/4269053715 >> v2:172.21.15.110:3300/0 conn(0x7f85d00a9c90 0x7f85d0143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.659+0000 7f85d4bd1640 1 -- 172.21.15.110:0/4269053715 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d00ac3c0 con 0x7f85d00aaaf0 2024-01-27T03:43:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.661+0000 7f1120a74640 1 --2- 172.21.15.110:0/1633267756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 0x7f111c0b1b80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1114009d90 tx=0x7f111402be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=249d5b051b10874d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.662+0000 7fe61d4e2640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4680 0x7fe61813ddc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe608034040 tx=0x7fe6080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.662+0000 7fe60e7fc640 10 client.?.objecter ms_handle_connect 0x7fe6180b4680 2024-01-27T03:43:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.662+0000 7fe60e7fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.663+0000 7f85d4bd1640 1 --2- 172.21.15.110:0/4269053715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d00aaaf0 0x7f85d01468d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f85d00adde0 tx=0x7f85c002b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=468f5f1e431f27a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.663+0000 7fae4b7fe640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c0ada30 0x7fae4c13bca0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fae44007210 tx=0x7fae44004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.663+0000 7fae48ff9640 10 client.?.objecter ms_handle_connect 0x7fae4c0ada30 2024-01-27T03:43:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.663+0000 7fae48ff9640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae48ff9640 1 -- 172.21.15.110:0/2777553887 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44008240 con 0x7fae4c0ada30 2024-01-27T03:43:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae48ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae48ff9640 10 monclient: dump: 2024-01-27T03:43:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae48ff9640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae48ff9640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae5132e640 5 monclient: authenticate success, global_id 4333 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae5132e640 10 monclient: _renew_subs 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae5132e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4c13e090 con 0x7fae4c0ada30 2024-01-27T03:43:32.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae5132e640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae5132e640 10 monclient: _renew_subs 2024-01-27T03:43:32.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae5132e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c13ebe0 con 0x7fae4c0ada30 2024-01-27T03:43:32.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.667+0000 7fae5132e640 1 librados: init done 2024-01-27T03:43:32.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7f85ceffd640 1 -- 172.21.15.110:0/4269053715 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c00317f0 con 0x7f85d00aaaf0 2024-01-27T03:43:32.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7f85ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7f85ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7f85ceffd640 10 monclient: dump: 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7f85ceffd640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7f85ceffd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7f85ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7f85d53d2640 5 monclient: authenticate success, global_id 4330 2024-01-27T03:43:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7f85d53d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7fae5132e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7fae5132e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7fae5132e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.668+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4c1371a0 con 0x7fae4c0ada30 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 1 -- 172.21.15.110:0/1633267756 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111402cc00 con 0x7f111c142eb0 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 10 monclient: dump: 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.688 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 1 -- 172.21.15.110:0/1633267756 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111402cd60 con 0x7f111c142eb0 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f11224d7640 5 monclient: authenticate success, global_id 4292 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f11224d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 1 -- 172.21.15.110:0/1633267756 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1114034660 con 0x7f111c142eb0 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:32.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f111affd640 10 monclient: dump: 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f11224d7640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.671+0000 7f11224d7640 4 set_mon_vals no callback set 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 10 monclient: shutdown 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 1 -- 172.21.15.110:0/1633267756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 msgr2=0x7f111c0b1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 1 --2- 172.21.15.110:0/1633267756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 0x7f111c0b1b80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1114009d90 tx=0x7f111402be50 comp rx=0 tx=0).stop 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 1 -- 172.21.15.110:0/1633267756 shutdown_connections 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 1 --2- 172.21.15.110:0/1633267756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 0x7f111c0b1b80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 1 --2- 172.21.15.110:0/1633267756 >> v2:172.21.15.110:3300/0 conn(0x7f111c0b4680 0x7f111c142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 1 -- 172.21.15.110:0/1633267756 >> 172.21.15.110:0/1633267756 conn(0x7f111c066490 msgr2=0x7f111c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 1 -- 172.21.15.110:0/1633267756 shutdown_connections 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.672+0000 7f11224d7640 1 -- 172.21.15.110:0/1633267756 wait complete. 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.673+0000 7f11224d7640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.673+0000 7f11224d7640 10 monclient: monmap: 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.691 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.673+0000 7f11224d7640 1 librados: starting msgr at 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.673+0000 7f11224d7640 1 librados: starting objecter 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 Processor -- start 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 -- start start 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 librados: setting wanted keys 2024-01-27T03:43:32.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 librados: calling monclient init 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 10 monclient: init 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0b4680 0x7f111c14bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 10 monclient(hunting): picked mon.noname-a con 0x7f111c0b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 0x7f111c1df0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 10 monclient(hunting): picked mon.noname-c con 0x7f111c142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f111c0ada00 0x7f111c1df5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 10 monclient(hunting): picked mon.noname-b con 0x7f111c0ada00 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c071760 con 0x7f111c0b4680 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c1dfb20 con 0x7f111c0ada00 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.674+0000 7f11224d7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c071bf0 con 0x7f111c142eb0 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.675+0000 7f11224d7640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.675+0000 7f11224d7640 10 monclient(hunting): authenticate will time out at 1496.028969s 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.675+0000 7f111bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0b4680 0x7f111c14bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7f111bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0b4680 0x7f111c14bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50148/0 (socket says 172.21.15.110:50148) 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7f111bfff640 1 -- 172.21.15.110:0/1056407926 learned_addr learned my addr 172.21.15.110:0/1056407926 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7f111bfff640 10 monclient(hunting): get_auth_request con 0x7f111c0b4680 auth_method 0 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7f111bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7f111bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7f111bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe21413e640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe21413e640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe21413e640 10 monclient: monmap: 2024-01-27T03:43:32.695 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.695 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.695 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.677666+0000 2024-01-27T03:43:32.695 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.677666+0000 2024-01-27T03:43:32.695 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.695 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.696 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.696 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.696 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.696 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.696 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.696 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 1 -- 172.21.15.110:0/3525405195 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6080328e0 con 0x7fe6180b4680 2024-01-27T03:43:32.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 10 monclient: dump: 2024-01-27T03:43:32.698 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.698 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.698 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 1 -- 172.21.15.110:0/3525405195 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe608032a80 con 0x7fe6180b4680 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 1 -- 172.21.15.110:0/3525405195 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608032e30 con 0x7fe6180b4680 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe60e7fc640 10 monclient: dump: 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.676+0000 7fe61ff47640 5 monclient: authenticate success, global_id 4289 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe61ff47640 10 monclient: _renew_subs 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe60f7fe640 4 set_mon_vals no callback set 2024-01-27T03:43:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe61ff47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6181d7910 con 0x7fe6180b4680 2024-01-27T03:43:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe61ff47640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe61ff47640 10 monclient: _renew_subs 2024-01-27T03:43:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe61ff47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6181d7d10 con 0x7fe6180b4680 2024-01-27T03:43:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe61ff47640 1 librados: init done 2024-01-27T03:43:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7f111bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7f111bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7f111bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe21413e640 1 Processor -- start 2024-01-27T03:43:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe21413e640 1 -- start start 2024-01-27T03:43:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.677+0000 7fe21413e640 10 monclient: init 2024-01-27T03:43:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe61ff47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe61ff47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe61ff47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61814cad0 con 0x7fe6180b4680 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe20c0b4680 0x7fe20c142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 10 monclient(hunting): picked mon.noname-a con 0x7fe20c0b4680 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe20c0b4a60 0x7fe20c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 10 monclient(hunting): picked mon.noname-b con 0x7fe20c0b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe20c142eb0 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c0afbe0 con 0x7fe20c142eb0 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20c142240 con 0x7fe20c0b4680 2024-01-27T03:43:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c0b0890 con 0x7fe20c0b4a60 2024-01-27T03:43:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.678+0000 7fe21413e640 10 monclient(hunting): authenticate will time out at 1496.032879s 2024-01-27T03:43:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-01-27T03:43:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 1 -- 172.21.15.110:0/1056407926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 msgr2=0x7f111c1df0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 0x7f111c1df0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 1 -- 172.21.15.110:0/1056407926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f111c0ada00 msgr2=0x7f111c1df5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f111c0ada00 0x7f111c1df5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 1 monclient: found mon.noname-a 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.684+0000 7f111bfff640 1 -- 172.21.15.110:0/1056407926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1114009a70 con 0x7f111c0b4680 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.687+0000 7f111bfff640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0b4680 0x7f111c14bd40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f111000e9e0 tx=0x7f111000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.687+0000 7f1118ff9640 10 client.?.objecter ms_handle_connect 0x7f111c0b4680 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.687+0000 7f1118ff9640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7f85ceffd640 1 -- 172.21.15.110:0/4269053715 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c0031950 con 0x7f85d00aaaf0 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7f85ceffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7f85ceffd640 1 -- 172.21.15.110:0/4269053715 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c0031ce0 con 0x7f85d00aaaf0 2024-01-27T03:43:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7f85ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7f85ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7f85ceffd640 10 monclient: dump: 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.708 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7f85d53d2640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7f85d53d2640 4 set_mon_vals no callback set 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7fae48ff9640 1 -- 172.21.15.110:0/2777553887 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae44007260 con 0x7fae4c0ada30 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7fae48ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7fae48ff9640 1 -- 172.21.15.110:0/2777553887 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44039790 con 0x7fae4c0ada30 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7fae48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7fae48ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7fae48ff9640 10 monclient: dump: 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.710 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.690+0000 7fae4a7fc640 4 set_mon_vals no callback set 2024-01-27T03:43:32.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.691+0000 7f85d53d2640 10 monclient: shutdown 2024-01-27T03:43:32.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.691+0000 7f85d53d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.691+0000 7f85d53d2640 1 -- 172.21.15.110:0/4269053715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d00aaaf0 msgr2=0x7f85d01468d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.691+0000 7f85d53d2640 1 --2- 172.21.15.110:0/4269053715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d00aaaf0 0x7f85d01468d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f85d00adde0 tx=0x7f85c002b890 comp rx=0 tx=0).stop 2024-01-27T03:43:32.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.691+0000 7fae48ff9640 1 -- 172.21.15.110:0/2777553887 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fae440083a0 con 0x7fae4c0ada30 2024-01-27T03:43:32.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.691+0000 7f85d53d2640 1 -- 172.21.15.110:0/4269053715 shutdown_connections 2024-01-27T03:43:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7fe2126db640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe20c142eb0 0x7fe20c0b19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55446/0 (socket says 172.21.15.110:55446) 2024-01-27T03:43:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7fe2126db640 1 -- 172.21.15.110:0/835355900 learned_addr learned my addr 172.21.15.110:0/835355900 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.691+0000 7f85d53d2640 1 --2- 172.21.15.110:0/4269053715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d00aaaf0 0x7f85d01468d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.691+0000 7f85d53d2640 1 --2- 172.21.15.110:0/4269053715 >> v2:172.21.15.110:3300/0 conn(0x7f85d00a9c90 0x7f85d0143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.691+0000 7f85d53d2640 1 --2- 172.21.15.110:0/4269053715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85d00a98b0 0x7f85d0143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7f85d53d2640 1 -- 172.21.15.110:0/4269053715 >> 172.21.15.110:0/4269053715 conn(0x7f85d00b3750 msgr2=0x7f85d00b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7fae48ff9640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fae3806c9a0 0x7fae3806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7f85d53d2640 1 -- 172.21.15.110:0/4269053715 shutdown_connections 2024-01-27T03:43:32.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7f85d53d2640 1 -- 172.21.15.110:0/4269053715 wait complete. 2024-01-27T03:43:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7fae4bfff640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fae3806c9a0 0x7fae3806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7fae4bfff640 10 monclient: get_auth_request con 0x7fae3806c9a0 auth_method 0 2024-01-27T03:43:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7fae4bfff640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fae3806c9a0 0x7fae3806ee60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00a750 tx=0x7fae3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.692+0000 7fae48ff9640 10 client.4333.objecter ms_handle_connect 0x7fae3806c9a0 2024-01-27T03:43:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.693+0000 7f85d53d2640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.693+0000 7f85d53d2640 10 monclient: monmap: 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.693+0000 7f85d53d2640 1 librados: starting msgr at 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.693+0000 7f85d53d2640 1 librados: starting objecter 2024-01-27T03:43:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.693+0000 7f85d53d2640 1 Processor -- start 2024-01-27T03:43:32.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.693+0000 7fe211eda640 1 --2- 172.21.15.110:0/835355900 >> v2:172.21.15.110:3300/0 conn(0x7fe20c0b4680 0x7fe20c142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f85d53d2640 1 -- start start 2024-01-27T03:43:32.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f85d53d2640 1 librados: setting wanted keys 2024-01-27T03:43:32.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f85d53d2640 1 librados: calling monclient init 2024-01-27T03:43:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f85d53d2640 10 monclient: init 2024-01-27T03:43:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f85d53d2640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f85d53d2640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T03:43:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f1118ff9640 1 -- 172.21.15.110:0/1056407926 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111000c9d0 con 0x7f111c0b4680 2024-01-27T03:43:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f1118ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f85d53d2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85d00a98b0 0x7f85d006df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f1118ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.694+0000 7f1118ff9640 10 monclient: dump: 2024-01-27T03:43:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f1118ff9640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f1118ff9640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f1118ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 10 monclient(hunting): picked mon.noname-a con 0x7f85d00a98b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85d00a9c90 0x7f85d0070780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 10 monclient(hunting): picked mon.noname-c con 0x7f85d00a9c90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d00aaaf0 0x7f85d0070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f11224d7640 5 monclient: authenticate success, global_id 4341 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f11224d7640 10 monclient: _renew_subs 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f11224d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c0ae4a0 con 0x7f111c0b4680 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f11224d7640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f11224d7640 10 monclient: _renew_subs 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f11224d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c071320 con 0x7f111c0b4680 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f11224d7640 1 librados: init done 2024-01-27T03:43:32.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 10 monclient(hunting): picked mon.noname-b con 0x7f85d00aaaf0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f1118ff9640 1 -- 172.21.15.110:0/1056407926 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111000cb30 con 0x7f111c0b4680 2024-01-27T03:43:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f1118ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f1118ff9640 1 -- 172.21.15.110:0/1056407926 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111000cee0 con 0x7f111c0b4680 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f1118ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f1118ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f1118ff9640 10 monclient: dump: 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.723 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d006fda0 con 0x7f85d00a98b0 2024-01-27T03:43:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d0071200 con 0x7f85d00aaaf0 2024-01-27T03:43:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d006fac0 con 0x7f85d00a9c90 2024-01-27T03:43:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f85d53d2640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.696+0000 7f85d53d2640 10 monclient(hunting): authenticate will time out at 1496.049977s 2024-01-27T03:43:32.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.695+0000 7f111a7fc640 4 set_mon_vals no callback set 2024-01-27T03:43:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.696+0000 7f11224d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.696+0000 7f11224d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.696+0000 7f11224d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.696+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc005080 con 0x7f111c0b4680 2024-01-27T03:43:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.697+0000 7fe211eda640 10 monclient(hunting): get_auth_request con 0x7fe20c0b4680 auth_method 0 2024-01-27T03:43:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.697+0000 7fe211eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.697+0000 7fe211eda640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.697+0000 7fe211eda640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.698+0000 7fe211eda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.698+0000 7fe211eda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.698+0000 7fe211eda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.698+0000 7fe210ed8640 1 -- 172.21.15.110:0/835355900 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57004005 0 0) 0x7fe20c0afbe0 con 0x7fe20c142eb0 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.698+0000 7fe210ed8640 10 monclient(hunting): _negotiate 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.698+0000 7fe210ed8640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.698+0000 7fe210ed8640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.698+0000 7fe210ed8640 10 monclient(hunting): my global_id is 4344 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.698+0000 7fe210ed8640 1 -- 172.21.15.110:0/835355900 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f8003520 con 0x7fe20c142eb0 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.701+0000 7fe211eda640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.701+0000 7fe211eda640 1 -- 172.21.15.110:0/835355900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe20c0b4a60 msgr2=0x7fe20c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.701+0000 7fe211eda640 1 --2- 172.21.15.110:0/835355900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe20c0b4a60 0x7fe20c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.701+0000 7fe211eda640 1 -- 172.21.15.110:0/835355900 >> v1:172.21.15.110:6789/0 conn(0x7fe20c142eb0 legacy=0x7fe20c0b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.701+0000 7fe211eda640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.701+0000 7fe211eda640 1 monclient: found mon.noname-a 2024-01-27T03:43:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.701+0000 7fe211eda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.701+0000 7fe211eda640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.701+0000 7fe211eda640 1 -- 172.21.15.110:0/835355900 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c0b0b00 con 0x7fe20c0b4680 2024-01-27T03:43:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.702+0000 7f85cffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85d00a98b0 0x7f85d006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.707+0000 7f85cffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85d00a98b0 0x7f85d006df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50162/0 (socket says 172.21.15.110:50162) 2024-01-27T03:43:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.707+0000 7f85cffff640 1 -- 172.21.15.110:0/3234982871 learned_addr learned my addr 172.21.15.110:0/3234982871 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.707+0000 7f85cffff640 10 monclient(hunting): get_auth_request con 0x7f85d00a98b0 auth_method 0 2024-01-27T03:43:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.707+0000 7f85cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.707+0000 7f85cffff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.707+0000 7f85cffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.708+0000 7fe211eda640 1 --2- 172.21.15.110:0/835355900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c0b4680 0x7fe20c142970 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe2080097a0 tx=0x7fe20802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24d2b2fe76a7de6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.709+0000 7f85cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.709+0000 7f85cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.709+0000 7f85cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 1 -- 172.21.15.110:0/3234982871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85d00a9c90 msgr2=0x7f85d0070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85d00a9c90 0x7f85d0070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 1 -- 172.21.15.110:0/3234982871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d00aaaf0 msgr2=0x7f85d0070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d00aaaf0 0x7f85d0070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 1 monclient: found mon.noname-a 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.713+0000 7f85cffff640 1 -- 172.21.15.110:0/3234982871 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c00094d0 con 0x7f85d00a98b0 2024-01-27T03:43:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 1 -- 172.21.15.110:0/835355900 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208031780 con 0x7fe20c0b4680 2024-01-27T03:43:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 10 monclient: dump: 2024-01-27T03:43:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 1 -- 172.21.15.110:0/835355900 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2080318e0 con 0x7fe20c0b4680 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 1 -- 172.21.15.110:0/835355900 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208031c70 con 0x7fe20c0b4680 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe210ed8640 10 monclient: dump: 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe21413e640 5 monclient: authenticate success, global_id 4347 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe21413e640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7fe21413e640 4 set_mon_vals no callback set 2024-01-27T03:43:32.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7f85cffff640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85d00a98b0 0x7f85d006df00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f85c800ba50 tx=0x7f85c800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7f85ccff9640 10 client.?.objecter ms_handle_connect 0x7f85d00a98b0 2024-01-27T03:43:32.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.718+0000 7f85ccff9640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 10 monclient: shutdown 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 1 -- 172.21.15.110:0/835355900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c0b4680 msgr2=0x7fe20c142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 1 --2- 172.21.15.110:0/835355900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c0b4680 0x7fe20c142970 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe2080097a0 tx=0x7fe20802b820 comp rx=0 tx=0).stop 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 1 -- 172.21.15.110:0/835355900 shutdown_connections 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 1 --2- 172.21.15.110:0/835355900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe20c0b4a60 0x7fe20c0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 1 --2- 172.21.15.110:0/835355900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c0b4680 0x7fe20c142970 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe2080097a0 tx=0x7fe20802b820 comp rx=0 tx=0).stop 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 1 -- 172.21.15.110:0/835355900 >> 172.21.15.110:0/835355900 conn(0x7fe20c066490 msgr2=0x7fe20c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 1 -- 172.21.15.110:0/835355900 shutdown_connections 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.719+0000 7fe21413e640 1 -- 172.21.15.110:0/835355900 wait complete. 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.720+0000 7fe21413e640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.720+0000 7fe21413e640 10 monclient: monmap: 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.720+0000 7fb77c3c7640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.720+0000 7fb77c3c7640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.720+0000 7fb77c3c7640 10 monclient: monmap: 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.721765+0000 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.721765+0000 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.738 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fe21413e640 1 librados: starting msgr at 2024-01-27T03:43:32.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fe21413e640 1 librados: starting objecter 2024-01-27T03:43:32.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fe21413e640 1 Processor -- start 2024-01-27T03:43:32.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fb77c3c7640 1 Processor -- start 2024-01-27T03:43:32.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fe21413e640 1 -- start start 2024-01-27T03:43:32.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fe21413e640 1 librados: setting wanted keys 2024-01-27T03:43:32.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fe21413e640 1 librados: calling monclient init 2024-01-27T03:43:32.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fe21413e640 10 monclient: init 2024-01-27T03:43:32.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fb77c3c7640 1 -- start start 2024-01-27T03:43:32.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.721+0000 7fb77c3c7640 10 monclient: init 2024-01-27T03:43:32.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T03:43:32.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe20c142eb0 0x7fe20c0a9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 10 monclient(hunting): picked mon.noname-c con 0x7fe20c142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe20c06ea40 0x7fe20c071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-01-27T03:43:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 10 monclient(hunting): picked mon.noname-b con 0x7fe20c06ea40 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c1dbd90 0x7fe20c1dc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7740b4680 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb7740b4a60 0x7fb7740b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 10 monclient(hunting): picked mon.noname-c con 0x7fb7740b4a60 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7740b3b30 0x7fb7740b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 10 monclient(hunting): picked mon.noname-b con 0x7fb7740b3b30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 10 monclient(hunting): picked mon.noname-a con 0x7fe20c1dbd90 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7740afbe0 con 0x7fb7740b4680 2024-01-27T03:43:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb774142240 con 0x7fb7740b4a60 2024-01-27T03:43:32.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740b0890 con 0x7fb7740b3b30 2024-01-27T03:43:32.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fb77c3c7640 10 monclient(hunting): authenticate will time out at 1496.076644s 2024-01-27T03:43:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c0aa120 con 0x7fe20c1dbd90 2024-01-27T03:43:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c0ab0d0 con 0x7fe20c06ea40 2024-01-27T03:43:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.722+0000 7fe21413e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.723+0000 7fe21413e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c1db870 con 0x7fe20c142eb0 2024-01-27T03:43:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.723+0000 7fe21413e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.723+0000 7fe21413e640 10 monclient(hunting): authenticate will time out at 1496.077120s 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.771+0000 7f3028851640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.771+0000 7f3028851640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.771+0000 7f3028851640 10 monclient: monmap: 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.773022+0000 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.773022+0000 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.774 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.772+0000 7f1f5fe75640 10 monclient: get_monmap_and_config 2024-01-27T03:43:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.772+0000 7f1f5fe75640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.772+0000 7f1f5fe75640 10 monclient: monmap: 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:32.773186+0000 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:32.773186+0000 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:32.775 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.772+0000 7f3028851640 1 Processor -- start 2024-01-27T03:43:32.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.772+0000 7f3028851640 1 -- start start 2024-01-27T03:43:32.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.772+0000 7f1f5fe75640 1 Processor -- start 2024-01-27T03:43:32.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.772+0000 7f3028851640 10 monclient: init 2024-01-27T03:43:32.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.772+0000 7f1f5fe75640 1 -- start start 2024-01-27T03:43:32.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.772+0000 7f1f5fe75640 10 monclient: init 2024-01-27T03:43:32.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-01-27T03:43:32.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30240b4680 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b4a60 0x7f30240b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-01-27T03:43:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1f5806fc00 0x7f1f5813d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 10 monclient(hunting): picked mon.noname-b con 0x7f30240b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f30240b3b30 0x7f30240b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 10 monclient(hunting): picked mon.noname-c con 0x7f30240b3b30 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 10 monclient(hunting): picked mon.noname-c con 0x7f1f5806fc00 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f5813d7a0 0x7f1f581403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 10 monclient(hunting): picked mon.noname-b con 0x7f1f5813d7a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30240afbe0 con 0x7f30240b4680 2024-01-27T03:43:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3024142240 con 0x7f30240b3b30 2024-01-27T03:43:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30240b0890 con 0x7f30240b4a60 2024-01-27T03:43:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f3028851640 10 monclient(hunting): authenticate will time out at 1496.127523s 2024-01-27T03:43:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f581408f0 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f580b0460 con 0x7f1f581408f0 2024-01-27T03:43:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f5813dbe0 con 0x7f1f5806fc00 2024-01-27T03:43:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58071b70 con 0x7f1f5813d7a0 2024-01-27T03:43:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.773+0000 7f1f5fe75640 10 monclient(hunting): authenticate will time out at 1496.127710s 2024-01-27T03:43:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.926+0000 7fae48ff9640 1 -- 172.21.15.110:0/2777553887 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fae4407a260 con 0x7fae4c0ada30 2024-01-27T03:43:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.926+0000 7fae48ff9640 10 client.4333.objecter ms_dispatch 0x7fae4c000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.926+0000 7fae48ff9640 3 client.4333.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.926+0000 7fae48ff9640 3 client.4333.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.926+0000 7fae48ff9640 20 client.4333.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.936+0000 7fae48ff9640 1 -- 172.21.15.110:0/2777553887 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fae44076500 con 0x7fae4c0ada30 2024-01-27T03:43:32.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.936+0000 7fae48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.936+0000 7fae48ff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.940+0000 7fe60e7fc640 1 -- 172.21.15.110:0/3525405195 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fe6080443a0 con 0x7fe6180b4680 2024-01-27T03:43:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.940+0000 7fe60e7fc640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe5fc06ca20 0x7fe5fc06eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.940+0000 7fe61dce3640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe5fc06ca20 0x7fe5fc06eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.940+0000 7fe61dce3640 10 monclient: get_auth_request con 0x7fe5fc06ca20 auth_method 0 2024-01-27T03:43:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.941+0000 7fe61dce3640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe5fc06ca20 0x7fe5fc06eee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe614007630 tx=0x7fe614000f10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.941+0000 7fe60e7fc640 10 client.4289.objecter ms_handle_connect 0x7fe5fc06ca20 2024-01-27T03:43:32.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.942+0000 7fb779962640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb7740b4a60 0x7fb7740b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.943+0000 7fb779962640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb7740b4a60 0x7fb7740b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40748/0 (socket says 172.21.15.110:40748) 2024-01-27T03:43:32.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.943+0000 7fb779962640 1 -- 172.21.15.110:0/2723306036 learned_addr learned my addr 172.21.15.110:0/2723306036 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.943+0000 7fb779962640 10 monclient(hunting): get_auth_request con 0x7fb7740b4a60 auth_method 0 2024-01-27T03:43:32.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.943+0000 7fb779962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.943+0000 7fb779962640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.943+0000 7fb779962640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.943+0000 7fe2116d9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe20c06ea40 0x7fe20c071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.944+0000 7fb779962640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.944+0000 7fb779962640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.944+0000 7fb779962640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.945+0000 7fe2116d9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe20c06ea40 0x7fe20c071580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40750/0 (socket says 172.21.15.110:40750) 2024-01-27T03:43:32.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.945+0000 7fe2116d9640 1 -- 172.21.15.110:0/405864428 learned_addr learned my addr 172.21.15.110:0/405864428 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.945+0000 7fe2116d9640 10 monclient(hunting): get_auth_request con 0x7fe20c06ea40 auth_method 0 2024-01-27T03:43:32.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.945+0000 7fe2116d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.945+0000 7fe2116d9640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.945+0000 7fe2116d9640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.946+0000 7fb779962640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-01-27T03:43:32.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.946+0000 7fb779962640 1 -- 172.21.15.110:0/2723306036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7740b3b30 msgr2=0x7fb7740b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.946+0000 7fb779962640 1 --2- 172.21.15.110:0/2723306036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7740b3b30 0x7fb7740b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.946+0000 7fb779962640 1 -- 172.21.15.110:0/2723306036 >> v1:172.21.15.162:6789/0 conn(0x7fb7740b4680 legacy=0x7fb774142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.946+0000 7fb779962640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.946+0000 7fb779962640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.946+0000 7fb779962640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.946+0000 7fb779962640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.946+0000 7fb779962640 1 -- 172.21.15.110:0/2723306036 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7740b0b00 con 0x7fb7740b4a60 2024-01-27T03:43:32.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.949+0000 7fe2116d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.949+0000 7fe2116d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.949+0000 7fe2116d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.949+0000 7fb779962640 1 --2- 172.21.15.110:0/2723306036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4a60 0x7fb7740b1470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb764009870 tx=0x7fb76402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56221898d872c62d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-01-27T03:43:32.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 1 -- 172.21.15.110:0/405864428 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe20c142eb0 msgr2=0x7fe20c0a9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe20c142eb0 0x7fe20c0a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 1 -- 172.21.15.110:0/405864428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c1dbd90 msgr2=0x7fe20c1dc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c1dbd90 0x7fe20c1dc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 1 monclient: found mon.noname-b 2024-01-27T03:43:32.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.952+0000 7fe2116d9640 1 -- 172.21.15.110:0/405864428 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe208009400 con 0x7fe20c06ea40 2024-01-27T03:43:32.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.956+0000 7f1f5dc11640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1f5806fc00 0x7f1f5813d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.957+0000 7f1f5dc11640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1f5806fc00 0x7f1f5813d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40772/0 (socket says 172.21.15.110:40772) 2024-01-27T03:43:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.957+0000 7f1f5dc11640 1 -- 172.21.15.110:0/2952931328 learned_addr learned my addr 172.21.15.110:0/2952931328 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.957+0000 7f1f5dc11640 10 monclient(hunting): get_auth_request con 0x7f1f5806fc00 auth_method 0 2024-01-27T03:43:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.957+0000 7f1f5dc11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.957+0000 7f1f5dc11640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.957+0000 7f1f5dc11640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.957+0000 7fe2116d9640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe20c06ea40 0x7fe20c071580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe20400b9f0 tx=0x7fe20400beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.957+0000 7fe2027fc640 10 client.?.objecter ms_handle_connect 0x7fe20c06ea40 2024-01-27T03:43:32.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.957+0000 7fe2027fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:32.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.958+0000 7f3022d9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f30240b3b30 0x7f30240b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.959+0000 7f1f5dc11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.959+0000 7f1f5dc11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.959+0000 7f1f5dc11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.959+0000 7f3022d9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f30240b3b30 0x7f30240b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40766/0 (socket says 172.21.15.110:40766) 2024-01-27T03:43:32.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.959+0000 7f3022d9d640 1 -- 172.21.15.110:0/549893724 learned_addr learned my addr 172.21.15.110:0/549893724 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.959+0000 7f3022d9d640 10 monclient(hunting): get_auth_request con 0x7f30240b3b30 auth_method 0 2024-01-27T03:43:32.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.959+0000 7f3022d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.959+0000 7f3022d9d640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.959+0000 7f3022d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.962+0000 7fe60e7fc640 1 -- 172.21.15.110:0/3525405195 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe60807a9b0 con 0x7fe6180b4680 2024-01-27T03:43:32.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.962+0000 7fe60e7fc640 10 client.4289.objecter ms_dispatch 0x7fe618000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:32.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.962+0000 7fe60e7fc640 3 client.4289.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:32.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.962+0000 7fe60e7fc640 3 client.4289.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:32.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7fe60e7fc640 20 client.4289.objecter dump_active .. 0 homeless 2024-01-27T03:43:32.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7fe60e7fc640 1 -- 172.21.15.110:0/3525405195 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fe608076c50 con 0x7fe6180b4680 2024-01-27T03:43:32.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7fe60e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7fe60e7fc640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:32.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7f1f5dc11640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2024-01-27T03:43:32.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7f1f5dc11640 1 -- 172.21.15.110:0/2952931328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f5813d7a0 msgr2=0x7f1f581403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7f1f5dc11640 1 --2- 172.21.15.110:0/2952931328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f5813d7a0 0x7f1f581403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7f1f5dc11640 1 -- 172.21.15.110:0/2952931328 >> v1:172.21.15.162:6789/0 conn(0x7f1f581408f0 legacy=0x7f1f581434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7f1f5dc11640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7f1f5dc11640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7f1f5dc11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7f1f5dc11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.963+0000 7f1f5dc11640 1 -- 172.21.15.110:0/2952931328 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f58071e50 con 0x7f1f5806fc00 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 1 -- 172.21.15.110:0/2723306036 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7640317c0 con 0x7fb7740b4a60 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 10 monclient: dump: 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.967 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 1 -- 172.21.15.110:0/2723306036 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb764031920 con 0x7fb7740b4a60 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb77c3c7640 5 monclient: authenticate success, global_id 4336 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb77c3c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb77c3c7640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 1 -- 172.21.15.110:0/2723306036 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764031cb0 con 0x7fb7740b4a60 2024-01-27T03:43:32.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb77c3c7640 4 set_mon_vals no callback set 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb779161640 10 monclient: dump: 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.969 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb77c3c7640 10 monclient: shutdown 2024-01-27T03:43:32.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb77c3c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:32.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb77c3c7640 1 -- 172.21.15.110:0/2723306036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4a60 msgr2=0x7fb7740b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.966+0000 7fb77c3c7640 1 --2- 172.21.15.110:0/2723306036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4a60 0x7fb7740b1470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb764009870 tx=0x7fb76402b860 comp rx=0 tx=0).stop 2024-01-27T03:43:32.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7fb77c3c7640 1 -- 172.21.15.110:0/2723306036 shutdown_connections 2024-01-27T03:43:32.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7fb77c3c7640 1 --2- 172.21.15.110:0/2723306036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7740b3b30 0x7fb7740b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7fb77c3c7640 1 --2- 172.21.15.110:0/2723306036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4a60 0x7fb7740b1470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7fb77c3c7640 1 -- 172.21.15.110:0/2723306036 >> 172.21.15.110:0/2723306036 conn(0x7fb774066490 msgr2=0x7fb7740668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:32.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7fb77c3c7640 1 -- 172.21.15.110:0/2723306036 shutdown_connections 2024-01-27T03:43:32.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7fb77c3c7640 1 -- 172.21.15.110:0/2723306036 wait complete. 2024-01-27T03:43:32.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7f3022d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7f3022d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7f3022d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7fb77c3c7640 10 monclient: build_initial_monmap 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.967+0000 7fb77c3c7640 10 monclient: monmap: 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.968+0000 7fb77c3c7640 1 librados: starting msgr at 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.968+0000 7fb77c3c7640 1 librados: starting objecter 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.968+0000 7fb77c3c7640 1 Processor -- start 2024-01-27T03:43:32.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.968+0000 7fb77c3c7640 1 -- start start 2024-01-27T03:43:32.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.968+0000 7fb77c3c7640 1 librados: setting wanted keys 2024-01-27T03:43:32.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.968+0000 7fb77c3c7640 1 librados: calling monclient init 2024-01-27T03:43:32.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.968+0000 7fb77c3c7640 10 monclient: init 2024-01-27T03:43:32.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.968+0000 7f1f5dc11640 1 --2- 172.21.15.110:0/2952931328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f5806fc00 0x7f1f5813d260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009870 tx=0x7f1f5402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42d1cb90abd11468 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:32.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7740b3b30 0x7fb77413e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient(hunting): picked mon.noname-a con 0x7fb7740b3b30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4680 0x7fb7740aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient(hunting): picked mon.noname-b con 0x7fb7740b4680 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7740ad4e0 0x7fb7740ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient(hunting): picked mon.noname-c con 0x7fb7740ad4e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740acb70 con 0x7fb7740b3b30 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740ab160 con 0x7fb7740b4680 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:32.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740acd70 con 0x7fb7740ad4e0 2024-01-27T03:43:32.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:32.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7fb77c3c7640 10 monclient(hunting): authenticate will time out at 1496.323407s 2024-01-27T03:43:32.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7f3022d9d640 10 monclient(hunting): handle_auth_done global_id 4345 payload 274 2024-01-27T03:43:32.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7f3022d9d640 1 -- 172.21.15.110:0/549893724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b4a60 msgr2=0x7f30240b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.969+0000 7f3022d9d640 1 --2- 172.21.15.110:0/549893724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b4a60 0x7f30240b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.970+0000 7f3022d9d640 1 -- 172.21.15.110:0/549893724 >> v1:172.21.15.110:6789/0 conn(0x7f30240b4680 legacy=0x7f3024142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:32.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.970+0000 7f3022d9d640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.970+0000 7f3022d9d640 1 monclient: found mon.noname-c 2024-01-27T03:43:32.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.970+0000 7f3022d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.970+0000 7f3022d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.970+0000 7f3022d9d640 1 -- 172.21.15.110:0/549893724 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240b0b00 con 0x7f30240b3b30 2024-01-27T03:43:32.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.974+0000 7fe2027fc640 1 -- 172.21.15.110:0/405864428 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20400cbd0 con 0x7fe20c06ea40 2024-01-27T03:43:32.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.974+0000 7fe2027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:32.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 10 monclient: dump: 2024-01-27T03:43:32.980 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.980 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.980 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.980 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 1 -- 172.21.15.110:0/405864428 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20400cd30 con 0x7fe20c06ea40 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 1 -- 172.21.15.110:0/405864428 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20400cea0 con 0x7fe20c06ea40 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe21413e640 5 monclient: authenticate success, global_id 4339 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe203fff640 4 set_mon_vals no callback set 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe2027fc640 10 monclient: dump: 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.982 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe21413e640 10 monclient: _renew_subs 2024-01-27T03:43:32.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe21413e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20c071ac0 con 0x7fe20c06ea40 2024-01-27T03:43:32.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe21413e640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe21413e640 10 monclient: _renew_subs 2024-01-27T03:43:32.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe21413e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20c13af40 con 0x7fe20c06ea40 2024-01-27T03:43:32.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fe21413e640 1 librados: init done 2024-01-27T03:43:32.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.975+0000 7fb779962640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4680 0x7fb7740aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.976+0000 7fe21413e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.976+0000 7fe21413e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.976+0000 7fe21413e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.976+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20c04ea50 con 0x7fe20c06ea40 2024-01-27T03:43:32.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.977+0000 7fb779962640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4680 0x7fb7740aa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40782/0 (socket says 172.21.15.110:40782) 2024-01-27T03:43:32.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.977+0000 7fb779962640 1 -- 172.21.15.110:0/1939885679 learned_addr learned my addr 172.21.15.110:0/1939885679 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:32.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.977+0000 7fb779962640 10 monclient(hunting): get_auth_request con 0x7fb7740b4680 auth_method 0 2024-01-27T03:43:32.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.977+0000 7fb779962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.977+0000 7fb779962640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.977+0000 7fb779962640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.978+0000 7f3022d9d640 1 --2- 172.21.15.110:0/549893724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30240b3b30 0x7f30240b19b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f301c009e50 tx=0x7f301c02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c557122a99964e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:32.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.979+0000 7fb77a964640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7740ad4e0 0x7fb7740ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:32.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.979+0000 7fb779962640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.979+0000 7fb779962640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.979+0000 7fb779962640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.979+0000 7fb77a964640 10 monclient(hunting): get_auth_request con 0x7fb7740ad4e0 auth_method 0 2024-01-27T03:43:32.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.979+0000 7fb77a964640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:32.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.979+0000 7fb77a964640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:32.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.979+0000 7fb77a964640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:32.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 1 -- 172.21.15.110:0/3234982871 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c800cbe0 con 0x7f85d00a98b0 2024-01-27T03:43:32.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 10 monclient: dump: 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 10 monclient: _finish_auth 0 2024-01-27T03:43:32.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 10 monclient: _check_auth_tickets 2024-01-27T03:43:32.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:32.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 1 -- 172.21.15.110:0/3234982871 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c800cd60 con 0x7f85d00a98b0 2024-01-27T03:43:32.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:32.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 1 -- 172.21.15.110:0/3234982871 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8014680 con 0x7f85d00a98b0 2024-01-27T03:43:32.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:32.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:32.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ccff9640 10 monclient: dump: 2024-01-27T03:43:32.989 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:32.990 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:32.990 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.990 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:32.992 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:32.992 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:32.992 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:32.992 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:32.992 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:32.992 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:32.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85ce7fc640 4 set_mon_vals no callback set 2024-01-27T03:43:32.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7fb77a964640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:32.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7fb77a964640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:32.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85d53d2640 5 monclient: authenticate success, global_id 4350 2024-01-27T03:43:32.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85d53d2640 10 monclient: _renew_subs 2024-01-27T03:43:32.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85d53d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d0070080 con 0x7f85d00a98b0 2024-01-27T03:43:32.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7fb77a964640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:32.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85d53d2640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:32.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85d53d2640 10 monclient: _renew_subs 2024-01-27T03:43:32.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85d53d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d006f0d0 con 0x7f85d00a98b0 2024-01-27T03:43:32.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.981+0000 7f85d53d2640 1 librados: init done 2024-01-27T03:43:32.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.982+0000 7f85d53d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.982+0000 7f85d53d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:32.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.982+0000 7f85d53d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:32.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.982+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d006edf0 con 0x7f85d00a98b0 2024-01-27T03:43:32.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-01-27T03:43:32.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 1 -- 172.21.15.110:0/1939885679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7740ad4e0 msgr2=0x7fb7740ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7740ad4e0 0x7fb7740ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 1 -- 172.21.15.110:0/1939885679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7740b3b30 msgr2=0x7fb77413e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:32.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7740b3b30 0x7fb77413e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:32.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 10 monclient: _finish_hunting 0 2024-01-27T03:43:32.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 1 monclient: found mon.noname-b 2024-01-27T03:43:32.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:32.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:32.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7fb779962640 1 -- 172.21.15.110:0/1939885679 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7640094d0 con 0x7fb7740b4680 2024-01-27T03:43:33.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7f1f5cc0f640 1 -- 172.21.15.110:0/2952931328 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54031640 con 0x7f1f5806fc00 2024-01-27T03:43:33.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7f1f5cc0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7f1f5cc0f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.983+0000 7f1f5cc0f640 10 monclient: dump: 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5cc0f640 10 monclient: _finish_auth 0 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5cc0f640 10 monclient: _check_auth_tickets 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5cc0f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5cc0f640 1 -- 172.21.15.110:0/2952931328 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f540317a0 con 0x7f1f5806fc00 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5cc0f640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5cc0f640 1 -- 172.21.15.110:0/2952931328 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54031b30 con 0x7f1f5806fc00 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5fe75640 5 monclient: authenticate success, global_id 4342 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5fe75640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5fe75640 4 set_mon_vals no callback set 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5cc0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5cc0f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5cc0f640 10 monclient: dump: 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.002 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5fe75640 10 monclient: shutdown 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5fe75640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2952931328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f5806fc00 msgr2=0x7f1f5813d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.984+0000 7f1f5fe75640 1 --2- 172.21.15.110:0/2952931328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f5806fc00 0x7f1f5813d260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009870 tx=0x7f1f5402b8f0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.985+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2952931328 shutdown_connections 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.985+0000 7f1f5fe75640 1 --2- 172.21.15.110:0/2952931328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f5813d7a0 0x7f1f581403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.985+0000 7f1f5fe75640 1 --2- 172.21.15.110:0/2952931328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f5806fc00 0x7f1f5813d260 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.985+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2952931328 >> 172.21.15.110:0/2952931328 conn(0x7f1f58066f00 msgr2=0x7f1f580b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.985+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2952931328 shutdown_connections 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.985+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2952931328 wait complete. 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.986+0000 7f1f5fe75640 10 monclient: build_initial_monmap 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.986+0000 7f1f5fe75640 10 monclient: monmap: 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:33.004 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.986+0000 7f1f5fe75640 1 librados: starting msgr at 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.986+0000 7f1f5fe75640 1 librados: starting objecter 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.986+0000 7f1f5fe75640 1 Processor -- start 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.986+0000 7f1f5fe75640 1 -- start start 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.986+0000 7f1f5fe75640 1 librados: setting wanted keys 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.986+0000 7f1f5fe75640 1 librados: calling monclient init 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.986+0000 7f1f5fe75640 10 monclient: init 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.987+0000 7f1f5fe75640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.987+0000 7f1f5fe75640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f5806fc00 0x7f1f58135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 10 monclient(hunting): picked mon.noname-a con 0x7f1f5806fc00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f5813d7a0 0x7f1f580aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 10 monclient(hunting): picked mon.noname-b con 0x7f1f5813d7a0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f581408f0 0x7f1f580acc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 10 monclient(hunting): picked mon.noname-c con 0x7f1f581408f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f580ae7b0 con 0x7f1f5806fc00 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58136a20 con 0x7f1f5813d7a0 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f580ac480 con 0x7f1f581408f0 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.988+0000 7f1f5fe75640 10 monclient(hunting): authenticate will time out at 1496.342410s 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.990+0000 7f1f5e412640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f581408f0 0x7f1f580acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.990+0000 7f1f5e412640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f581408f0 0x7f1f580acc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38544/0 (socket says 172.21.15.110:38544) 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.990+0000 7f1f5e412640 1 -- 172.21.15.110:0/2532242959 learned_addr learned my addr 172.21.15.110:0/2532242959 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.990+0000 7f1f5e412640 10 monclient(hunting): get_auth_request con 0x7f1f581408f0 auth_method 0 2024-01-27T03:43:33.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.990+0000 7f1f5e412640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.990+0000 7f1f5e412640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.990+0000 7f1f5e412640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.991+0000 7f1f5e412640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.991+0000 7f1f5e412640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.991+0000 7f1f5e412640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 1 -- 172.21.15.110:0/2532242959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f5813d7a0 msgr2=0x7f1f580aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f5813d7a0 0x7f1f580aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 1 -- 172.21.15.110:0/2532242959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f5806fc00 msgr2=0x7f1f58135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f5806fc00 0x7f1f58135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 10 monclient: _finish_hunting 0 2024-01-27T03:43:33.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 1 monclient: found mon.noname-c 2024-01-27T03:43:33.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:33.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.995+0000 7f1f5e412640 1 -- 172.21.15.110:0/2532242959 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f540094d0 con 0x7f1f581408f0 2024-01-27T03:43:33.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.998+0000 7f1f5e412640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f581408f0 0x7f1f580acc80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000b990 tx=0x7f1f5000be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:33.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.998+0000 7f1f4e7fc640 10 client.?.objecter ms_handle_connect 0x7f1f581408f0 2024-01-27T03:43:33.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:32.998+0000 7f1f4e7fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:33.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 1 -- 172.21.15.110:0/2532242959 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50018440 con 0x7f1f581408f0 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 10 monclient: dump: 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.010 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 1 -- 172.21.15.110:0/2532242959 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f500185a0 con 0x7f1f581408f0 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f5fe75640 5 monclient: authenticate success, global_id 4298 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f5fe75640 10 monclient: _renew_subs 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f5fe75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f580ac150 con 0x7f1f581408f0 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4ffff640 4 set_mon_vals no callback set 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f5fe75640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:33.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f5fe75640 10 monclient: _renew_subs 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 1 -- 172.21.15.110:0/2532242959 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50018990 con 0x7f1f581408f0 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f5fe75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f580ad1c0 con 0x7f1f581408f0 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f5fe75640 1 librados: init done 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.003+0000 7f1f4e7fc640 10 monclient: dump: 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.012 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.004+0000 7f1f5fe75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.004+0000 7f1f5fe75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.004+0000 7f1f5fe75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.004+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f580709f0 con 0x7f1f581408f0 2024-01-27T03:43:33.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.014+0000 7f1118ff9640 1 -- 172.21.15.110:0/1056407926 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f1110013470 con 0x7f111c0b4680 2024-01-27T03:43:33.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.015+0000 7f1118ff9640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f110806ca20 0x7f110806eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.015+0000 7f111b7fe640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f110806ca20 0x7f110806eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:33.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.015+0000 7f111b7fe640 10 monclient: get_auth_request con 0x7f110806ca20 auth_method 0 2024-01-27T03:43:33.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.016+0000 7f1118ff9640 10 client.4341.objecter ms_handle_connect 0x7f110806ca20 2024-01-27T03:43:33.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.016+0000 7f111b7fe640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f110806ca20 0x7f110806eee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f111c066f80 tx=0x7f110c007a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:33.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.023+0000 7f1f4e7fc640 1 -- 172.21.15.110:0/2532242959 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f1f500213b0 con 0x7f1f581408f0 2024-01-27T03:43:33.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.024+0000 7f1f4e7fc640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f1f3c06c9a0 0x7f1f3c06ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.024+0000 7f1f5dc11640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f1f3c06c9a0 0x7f1f3c06ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:33.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.024+0000 7f1f5dc11640 10 monclient: get_auth_request con 0x7f1f3c06c9a0 auth_method 0 2024-01-27T03:43:33.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.024+0000 7f1f4e7fc640 10 client.4298.objecter ms_handle_connect 0x7f1f3c06c9a0 2024-01-27T03:43:33.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.024+0000 7f1f5dc11640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f1f3c06c9a0 0x7f1f3c06ee60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1f54036000 tx=0x7f1f54030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:33.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.027+0000 7f1f4e7fc640 1 -- 172.21.15.110:0/2532242959 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1f5005d600 con 0x7f1f581408f0 2024-01-27T03:43:33.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.027+0000 7f1f4e7fc640 10 client.4298.objecter ms_dispatch 0x7f1f58000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:33.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.027+0000 7f1f4e7fc640 3 client.4298.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:33.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.027+0000 7f1f4e7fc640 3 client.4298.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:33.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.027+0000 7f1f4e7fc640 20 client.4298.objecter dump_active .. 0 homeless 2024-01-27T03:43:33.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.031+0000 7f1118ff9640 1 -- 172.21.15.110:0/1056407926 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1110059500 con 0x7f111c0b4680 2024-01-27T03:43:33.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.031+0000 7f1118ff9640 10 client.4341.objecter ms_dispatch 0x7f111c000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:33.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.031+0000 7f1118ff9640 3 client.4341.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:33.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.031+0000 7f1118ff9640 3 client.4341.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:33.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.031+0000 7f1118ff9640 20 client.4341.objecter dump_active .. 0 homeless 2024-01-27T03:43:33.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.031+0000 7f1118ff9640 1 -- 172.21.15.110:0/1056407926 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f11100550a0 con 0x7f111c0b4680 2024-01-27T03:43:33.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.031+0000 7f1118ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.031+0000 7f1118ff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:33.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.152+0000 7fae5132e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-27T03:43:33.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.152+0000 7fae5132e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-27T03:43:33.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.152+0000 7fae5132e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:33.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.152+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fae4c1371a0 con 0x7fae4c0ada30 2024-01-27T03:43:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.174+0000 7fe61ff47640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-27T03:43:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.174+0000 7fe61ff47640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-27T03:43:33.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.174+0000 7fe61ff47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.174+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fe61814cad0 con 0x7fe6180b4680 2024-01-27T03:43:33.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 1 -- 172.21.15.110:0/549893724 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c02c9e0 con 0x7f30240b3b30 2024-01-27T03:43:33.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 10 monclient: dump: 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 10 monclient: _finish_auth 0 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 10 monclient: _check_auth_tickets 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 1 -- 172.21.15.110:0/549893724 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c02cb40 con 0x7f30240b3b30 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f3028851640 5 monclient: authenticate success, global_id 4345 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 1 -- 172.21.15.110:0/549893724 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c0343f0 con 0x7f30240b3b30 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f302159a640 10 monclient: dump: 2024-01-27T03:43:33.225 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f3028851640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:33.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7f3028851640 4 set_mon_vals no callback set 2024-01-27T03:43:33.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7fb779962640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4680 0x7fb7740aa9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb764009870 tx=0x7fb7640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:33.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7fb76affd640 10 client.?.objecter ms_handle_connect 0x7fb7740b4680 2024-01-27T03:43:33.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.222+0000 7fb76affd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:33.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 10 monclient: shutdown 2024-01-27T03:43:33.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 1 -- 172.21.15.110:0/549893724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30240b3b30 msgr2=0x7f30240b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 1 --2- 172.21.15.110:0/549893724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30240b3b30 0x7f30240b19b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f301c009e50 tx=0x7f301c02be40 comp rx=0 tx=0).stop 2024-01-27T03:43:33.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 1 -- 172.21.15.110:0/549893724 shutdown_connections 2024-01-27T03:43:33.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 1 --2- 172.21.15.110:0/549893724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30240b3b30 0x7f30240b19b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 1 --2- 172.21.15.110:0/549893724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b4a60 0x7f30240b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 1 -- 172.21.15.110:0/549893724 >> 172.21.15.110:0/549893724 conn(0x7f3024066490 msgr2=0x7f30240668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 1 -- 172.21.15.110:0/549893724 shutdown_connections 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.223+0000 7f3028851640 1 -- 172.21.15.110:0/549893724 wait complete. 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.224+0000 7f3028851640 10 monclient: build_initial_monmap 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.224+0000 7f3028851640 10 monclient: monmap: 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:33.230 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.224+0000 7f3028851640 1 librados: starting msgr at 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.224+0000 7f3028851640 1 librados: starting objecter 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.225+0000 7f3028851640 1 Processor -- start 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.225+0000 7f3028851640 1 -- start start 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.225+0000 7f3028851640 1 librados: setting wanted keys 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.225+0000 7f3028851640 1 librados: calling monclient init 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.225+0000 7f3028851640 10 monclient: init 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.225+0000 7f3028851640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.225+0000 7f3028851640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.225+0000 7f3028851640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b3b30 0x7f30240ab350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.225+0000 7f3028851640 10 monclient(hunting): picked mon.noname-c con 0x7f30240b3b30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30240b4680 0x7f302413aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 10 monclient(hunting): picked mon.noname-a con 0x7f30240b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:33.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30240a94b0 0x7f30240a9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 10 monclient(hunting): picked mon.noname-b con 0x7f30240a94b0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:33.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:33.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240aa6c0 con 0x7f30240b4680 2024-01-27T03:43:33.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:33.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240abec0 con 0x7f30240a94b0 2024-01-27T03:43:33.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:33.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30240ad120 con 0x7f30240b3b30 2024-01-27T03:43:33.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:33.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.226+0000 7f3028851640 10 monclient(hunting): authenticate will time out at 1496.580020s 2024-01-27T03:43:33.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.237+0000 7fe2027fc640 1 -- 172.21.15.110:0/405864428 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fe20401d3b0 con 0x7fe20c06ea40 2024-01-27T03:43:33.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.238+0000 7fe2027fc640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe1f006c9a0 0x7fe1f006ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.238+0000 7fe211eda640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe1f006c9a0 0x7fe1f006ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:33.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.238+0000 7fe211eda640 10 monclient: get_auth_request con 0x7fe1f006c9a0 auth_method 0 2024-01-27T03:43:33.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.238+0000 7fe211eda640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe1f006c9a0 0x7fe1f006ee60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe208036000 tx=0x7fe208030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:33.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.238+0000 7fe2027fc640 10 client.4339.objecter ms_handle_connect 0x7fe1f006c9a0 2024-01-27T03:43:33.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.239+0000 7fe2027fc640 1 -- 172.21.15.110:0/405864428 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe204058e60 con 0x7fe20c06ea40 2024-01-27T03:43:33.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.239+0000 7fe2027fc640 10 client.4339.objecter ms_dispatch 0x7fe20c000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:33.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.239+0000 7fe2027fc640 3 client.4339.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:33.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.239+0000 7fe2027fc640 3 client.4339.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:33.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.240+0000 7fe2027fc640 20 client.4339.objecter dump_active .. 0 homeless 2024-01-27T03:43:33.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.249+0000 7fe2027fc640 1 -- 172.21.15.110:0/405864428 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fe204054a00 con 0x7fe20c06ea40 2024-01-27T03:43:33.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.249+0000 7fe2027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.249+0000 7fe2027fc640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:33.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 1 -- 172.21.15.110:0/1939885679 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764032960 con 0x7fb7740b4680 2024-01-27T03:43:33.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 10 monclient: dump: 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 10 monclient: _finish_auth 0 2024-01-27T03:43:33.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 1 -- 172.21.15.110:0/1939885679 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb764032ac0 con 0x7fb7740b4680 2024-01-27T03:43:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb77c3c7640 5 monclient: authenticate success, global_id 4348 2024-01-27T03:43:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb77c3c7640 10 monclient: _renew_subs 2024-01-27T03:43:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb77c3c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:33.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7740ad050 con 0x7fb7740b4680 2024-01-27T03:43:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 1 -- 172.21.15.110:0/1939885679 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764032e70 con 0x7fb7740b4680 2024-01-27T03:43:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb76affd640 10 monclient: dump: 2024-01-27T03:43:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.259 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb77c3c7640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb77c3c7640 10 monclient: _renew_subs 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb77c3c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7740abd10 con 0x7fb7740b4680 2024-01-27T03:43:33.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb77c3c7640 1 librados: init done 2024-01-27T03:43:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.255+0000 7fb778960640 4 set_mon_vals no callback set 2024-01-27T03:43:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.257+0000 7fb77c3c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.257+0000 7fb77c3c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.257+0000 7fb77c3c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:33.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.257+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb774074d10 con 0x7fb7740b4680 2024-01-27T03:43:33.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.264+0000 7f3022d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30240a94b0 0x7f30240a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:33.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.284+0000 7f302259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b3b30 0x7f30240ab350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:33.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.294+0000 7f11224d7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-27T03:43:33.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.294+0000 7f11224d7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-27T03:43:33.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.294+0000 7f11224d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:33.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.294+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f10fc005600 con 0x7f111c0b4680 2024-01-27T03:43:33.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.302+0000 7f1f4e7fc640 1 -- 172.21.15.110:0/2532242959 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f1f500591a0 con 0x7f1f581408f0 2024-01-27T03:43:33.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.302+0000 7f1f4e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.302+0000 7f1f4e7fc640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:33.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.310+0000 7f302259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b3b30 0x7f30240ab350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38558/0 (socket says 172.21.15.110:38558) 2024-01-27T03:43:33.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.310+0000 7f302259c640 1 -- 172.21.15.110:0/2018620571 learned_addr learned my addr 172.21.15.110:0/2018620571 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:33.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.310+0000 7f302259c640 10 monclient(hunting): get_auth_request con 0x7f30240b3b30 auth_method 0 2024-01-27T03:43:33.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.310+0000 7f302259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:33.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.310+0000 7f302259c640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:33.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.310+0000 7f302259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:33.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.313+0000 7f302259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:33.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.313+0000 7f302259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:33.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.313+0000 7f302259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:33.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.314+0000 7fe60e7fc640 1 -- 172.21.15.110:0/3525405195 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe61814cad0 con 0x7fe6180b4680 2024-01-27T03:43:33.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.314+0000 7fe60e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-27T03:43:33.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.314+0000 7fe60e7fc640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:33.315 INFO:teuthology.orchestra.run.smithi110.stdout:55834574853 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 10 librados: watch_flush enter 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 10 librados: watch_flush exit 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 20 client.4289.objecter shutdown clearing up homeless session... 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 10 client.4289.objecter successfully canceled tick 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe5fc06ca20 msgr2=0x7fe5fc06eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe5fc06ca20 0x7fe5fc06eee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe614007630 tx=0x7fe614000f10 comp rx=0 tx=0).stop 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 10 monclient: shutdown 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4680 msgr2=0x7fe61813ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4680 0x7fe61813ddc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe608034040 tx=0x7fe6080046c0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 shutdown_connections 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe5fc06ca20 0x7fe5fc06eee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6180b4680 0x7fe61813ddc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61813d870 0x7fe6181d71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 --2- 172.21.15.110:0/3525405195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6180b3b30 0x7fe61813a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 >> 172.21.15.110:0/3525405195 conn(0x7fe618066490 msgr2=0x7fe6180b7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 shutdown_connections 2024-01-27T03:43:33.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 -- 172.21.15.110:0/3525405195 wait complete. 2024-01-27T03:43:33.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.316+0000 7fe61ff47640 1 librados: shutdown 2024-01-27T03:43:33.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.317+0000 7f302259c640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-01-27T03:43:33.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.317+0000 7f302259c640 1 -- 172.21.15.110:0/2018620571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30240a94b0 msgr2=0x7f30240a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.317+0000 7f302259c640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30240a94b0 0x7f30240a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.317+0000 7f302259c640 1 -- 172.21.15.110:0/2018620571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30240b4680 msgr2=0x7f302413aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.317+0000 7f302259c640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30240b4680 0x7f302413aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.318+0000 7f302259c640 10 monclient: _finish_hunting 0 2024-01-27T03:43:33.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.318+0000 7f302259c640 1 monclient: found mon.noname-c 2024-01-27T03:43:33.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.318+0000 7f302259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:33.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.318+0000 7f302259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.318+0000 7f302259c640 1 -- 172.21.15.110:0/2018620571 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c009b30 con 0x7f30240b3b30 2024-01-27T03:43:33.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.321+0000 7f302259c640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b3b30 0x7f30240ab350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f301400c410 tx=0x7f301400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:33.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.321+0000 7f3012ffd640 10 client.?.objecter ms_handle_connect 0x7f30240b3b30 2024-01-27T03:43:33.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.321+0000 7f3012ffd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:33.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-01-27T03:43:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 1 -- 172.21.15.110:0/2018620571 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30140035f0 con 0x7f30240b3b30 2024-01-27T03:43:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 10 monclient: dump: 2024-01-27T03:43:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.331 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 10 monclient: _finish_auth 0 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:33.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3028851640 5 monclient: authenticate success, global_id 4301 2024-01-27T03:43:33.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3028851640 10 monclient: _renew_subs 2024-01-27T03:43:33.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 1 -- 172.21.15.110:0/2018620571 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3014003750 con 0x7f30240b3b30 2024-01-27T03:43:33.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3028851640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30240a9ea0 con 0x7f30240b3b30 2024-01-27T03:43:33.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:33.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 1 -- 172.21.15.110:0/2018620571 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014003b40 con 0x7f30240b3b30 2024-01-27T03:43:33.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3012ffd640 10 monclient: dump: 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:33.334 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:33.335 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:33.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3028851640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:33.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3020d99640 4 set_mon_vals no callback set 2024-01-27T03:43:33.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3028851640 10 monclient: _renew_subs 2024-01-27T03:43:33.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3028851640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30240aabb0 con 0x7f30240b3b30 2024-01-27T03:43:33.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.329+0000 7f3028851640 1 librados: init done 2024-01-27T03:43:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.330+0000 7f3028851640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.330+0000 7f3028851640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.330+0000 7f3028851640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.330+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3024074d10 con 0x7f30240b3b30 2024-01-27T03:43:33.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.350+0000 7f3012ffd640 1 -- 172.21.15.110:0/2018620571 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f3014003ca0 con 0x7f30240b3b30 2024-01-27T03:43:33.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.350+0000 7f3012ffd640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f300406c9a0 0x7f300406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.351+0000 7f3021d9b640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f300406c9a0 0x7f300406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:33.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.351+0000 7f3021d9b640 10 monclient: get_auth_request con 0x7f300406c9a0 auth_method 0 2024-01-27T03:43:33.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.351+0000 7f3012ffd640 10 client.4301.objecter ms_handle_connect 0x7f300406c9a0 2024-01-27T03:43:33.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.351+0000 7f3021d9b640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f300406c9a0 0x7f300406ee60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f30240aca10 tx=0x7f3018006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:33.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.439+0000 7fe21413e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-27T03:43:33.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.439+0000 7fe21413e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-27T03:43:33.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.439+0000 7fe21413e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:33.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.439+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe20c069970 con 0x7fe20c06ea40 2024-01-27T03:43:33.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.454+0000 7f85ccff9640 1 -- 172.21.15.110:0/3234982871 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f85c801d3b0 con 0x7f85d00a98b0 2024-01-27T03:43:33.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.455+0000 7f85ccff9640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f85bc06c9a0 0x7f85bc06ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.455+0000 7f85cf7fe640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f85bc06c9a0 0x7f85bc06ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:33.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.455+0000 7f85cf7fe640 10 monclient: get_auth_request con 0x7f85bc06c9a0 auth_method 0 2024-01-27T03:43:33.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.456+0000 7f85cf7fe640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f85bc06c9a0 0x7f85bc06ee60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f85d01433c0 tx=0x7f85c4006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:33.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.456+0000 7f85ccff9640 10 client.4350.objecter ms_handle_connect 0x7f85bc06c9a0 2024-01-27T03:43:33.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.474+0000 7f85ccff9640 1 -- 172.21.15.110:0/3234982871 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f85c8058d10 con 0x7f85d00a98b0 2024-01-27T03:43:33.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.474+0000 7f85ccff9640 10 client.4350.objecter ms_dispatch 0x7f85d0000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:33.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.474+0000 7f85ccff9640 3 client.4350.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:33.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.474+0000 7f85ccff9640 3 client.4350.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:33.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.474+0000 7f85ccff9640 20 client.4350.objecter dump_active .. 0 homeless 2024-01-27T03:43:33.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.474+0000 7f85ccff9640 1 -- 172.21.15.110:0/3234982871 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f85c80548b0 con 0x7f85d00a98b0 2024-01-27T03:43:33.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.474+0000 7f85ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.474+0000 7f85ccff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:33.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.477+0000 7f1118ff9640 1 -- 172.21.15.110:0/1056407926 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1110017090 con 0x7f111c0b4680 2024-01-27T03:43:33.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.477+0000 7f1118ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-27T03:43:33.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.477+0000 7f1118ff9640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:33.478 INFO:teuthology.orchestra.run.smithi110.stdout:64424509445 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 10 librados: watch_flush enter 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 10 librados: watch_flush exit 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 20 client.4341.objecter shutdown clearing up homeless session... 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 10 client.4341.objecter successfully canceled tick 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f110806ca20 msgr2=0x7f110806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f110806ca20 0x7f110806eee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f111c066f80 tx=0x7f110c007a40 comp rx=0 tx=0).stop 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 10 monclient: shutdown 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0b4680 msgr2=0x7f111c14bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0b4680 0x7f111c14bd40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f111000e9e0 tx=0x7f111000eea0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 shutdown_connections 2024-01-27T03:43:33.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f110806ca20 0x7f110806eee0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c142eb0 0x7f111c1df0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f111c0ada00 0x7f111c1df5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 --2- 172.21.15.110:0/1056407926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0b4680 0x7f111c14bd40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 >> 172.21.15.110:0/1056407926 conn(0x7f111c066490 msgr2=0x7f111c13f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.480+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 shutdown_connections 2024-01-27T03:43:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.481+0000 7f11224d7640 1 -- 172.21.15.110:0/1056407926 wait complete. 2024-01-27T03:43:33.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.481+0000 7f11224d7640 1 librados: shutdown 2024-01-27T03:43:33.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.5 2024-01-27T03:43:33.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.513+0000 7fb76affd640 1 -- 172.21.15.110:0/1939885679 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fb76404e450 con 0x7fb7740b4680 2024-01-27T03:43:33.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.514+0000 7fb76affd640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.514+0000 7fb77a163640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.514+0000 7fb77a163640 10 monclient: get_auth_request con 0x7fb75806c9a0 auth_method 0 2024-01-27T03:43:33.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.514+0000 7fb77a163640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb770001560 tx=0x7fb770007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.514+0000 7fb76affd640 10 client.4348.objecter ms_handle_connect 0x7fb75806c9a0 2024-01-27T03:43:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.515+0000 7f1f5fe75640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-27T03:43:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.515+0000 7f1f5fe75640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-27T03:43:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.515+0000 7f1f5fe75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.515+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f1f58074d10 con 0x7f1f581408f0 2024-01-27T03:43:33.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.522+0000 7fb76affd640 1 -- 172.21.15.110:0/1939885679 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb76407afa0 con 0x7fb7740b4680 2024-01-27T03:43:33.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.522+0000 7fb76affd640 10 client.4348.objecter ms_dispatch 0x7fb774000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:33.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.522+0000 7fb76affd640 3 client.4348.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:33.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.522+0000 7fb76affd640 3 client.4348.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:33.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.522+0000 7fb76affd640 20 client.4348.objecter dump_active .. 0 homeless 2024-01-27T03:43:33.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.533+0000 7fb76affd640 1 -- 172.21.15.110:0/1939885679 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fb764076b40 con 0x7fb7740b4680 2024-01-27T03:43:33.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.533+0000 7fb76affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.533+0000 7fb76affd640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:33.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.533+0000 7fae48ff9640 1 -- 172.21.15.110:0/2777553887 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fae4c1371a0 con 0x7fae4c0ada30 2024-01-27T03:43:33.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.533+0000 7fae48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-27T03:43:33.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.533+0000 7fae48ff9640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:33.534 INFO:teuthology.orchestra.run.smithi110.stdout:85899345927 2024-01-27T03:43:33.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 10 librados: watch_flush enter 2024-01-27T03:43:33.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 10 librados: watch_flush exit 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 20 client.4333.objecter shutdown clearing up homeless session... 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 10 client.4333.objecter successfully canceled tick 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fae3806c9a0 msgr2=0x7fae3806ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fae3806c9a0 0x7fae3806ee60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00a750 tx=0x7fae3c008040 comp rx=0 tx=0).stop 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 10 monclient: shutdown 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c0ada30 msgr2=0x7fae4c13bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c0ada30 0x7fae4c13bca0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fae44007210 tx=0x7fae44004a40 comp rx=0 tx=0).stop 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 shutdown_connections 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fae3806c9a0 0x7fae3806ee60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae4c0aed60 0x7fae4c13c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae4c0ada30 0x7fae4c13bca0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 --2- 172.21.15.110:0/2777553887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae4c072a50 0x7fae4c1c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 >> 172.21.15.110:0/2777553887 conn(0x7fae4c0b3b70 msgr2=0x7fae4c0aa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 shutdown_connections 2024-01-27T03:43:33.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 -- 172.21.15.110:0/2777553887 wait complete. 2024-01-27T03:43:33.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.535+0000 7fae5132e640 1 librados: shutdown 2024-01-27T03:43:33.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345927 for osd.4 2024-01-27T03:43:33.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.597+0000 7fe2027fc640 1 -- 172.21.15.110:0/405864428 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe204019070 con 0x7fe20c06ea40 2024-01-27T03:43:33.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.597+0000 7fe2027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-27T03:43:33.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.597+0000 7fe2027fc640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:33.598 INFO:teuthology.orchestra.run.smithi110.stdout:85899345929 2024-01-27T03:43:33.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 10 librados: watch_flush enter 2024-01-27T03:43:33.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 10 librados: watch_flush exit 2024-01-27T03:43:33.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 20 client.4339.objecter shutdown clearing up homeless session... 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 10 client.4339.objecter successfully canceled tick 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe1f006c9a0 msgr2=0x7fe1f006ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe1f006c9a0 0x7fe1f006ee60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe208036000 tx=0x7fe208030000 comp rx=0 tx=0).stop 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 10 monclient: shutdown 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe20c06ea40 msgr2=0x7fe20c071580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe20c06ea40 0x7fe20c071580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe20400b9f0 tx=0x7fe20400beb0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 shutdown_connections 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe1f006c9a0 0x7fe1f006ee60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe20c1dbd90 0x7fe20c1dc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe20c06ea40 0x7fe20c071580 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 --2- 172.21.15.110:0/405864428 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe20c142eb0 0x7fe20c0a9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 >> 172.21.15.110:0/405864428 conn(0x7fe20c066490 msgr2=0x7fe20c0b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 shutdown_connections 2024-01-27T03:43:33.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 -- 172.21.15.110:0/405864428 wait complete. 2024-01-27T03:43:33.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.599+0000 7fe21413e640 1 librados: shutdown 2024-01-27T03:43:33.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345931 got 85899345929 for osd.6 2024-01-27T03:43:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.653+0000 7f3012ffd640 1 -- 172.21.15.110:0/2018620571 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f30140585c0 con 0x7f30240b3b30 2024-01-27T03:43:33.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.653+0000 7f3012ffd640 10 client.4301.objecter ms_dispatch 0x7f3024000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:33.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.653+0000 7f3012ffd640 3 client.4301.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:33.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.653+0000 7f3012ffd640 3 client.4301.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:33.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.653+0000 7f3012ffd640 20 client.4301.objecter dump_active .. 0 homeless 2024-01-27T03:43:33.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.653+0000 7f3012ffd640 1 -- 172.21.15.110:0/2018620571 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f3014054160 con 0x7f30240b3b30 2024-01-27T03:43:33.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.653+0000 7f3012ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:33.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.653+0000 7f3012ffd640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.665+0000 7f1f4e7fc640 1 -- 172.21.15.110:0/2532242959 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1f5001d020 con 0x7f1f581408f0 2024-01-27T03:43:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.665+0000 7f1f4e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-27T03:43:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.665+0000 7f1f4e7fc640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:33.666 INFO:teuthology.orchestra.run.smithi110.stdout:68719476742 2024-01-27T03:43:33.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.667+0000 7f1f5fe75640 10 librados: watch_flush enter 2024-01-27T03:43:33.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 10 librados: watch_flush exit 2024-01-27T03:43:33.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 20 client.4298.objecter shutdown clearing up homeless session... 2024-01-27T03:43:33.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 10 client.4298.objecter successfully canceled tick 2024-01-27T03:43:33.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f1f3c06c9a0 msgr2=0x7f1f3c06ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f1f3c06c9a0 0x7f1f3c06ee60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1f54036000 tx=0x7f1f54030000 comp rx=0 tx=0).stop 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 10 monclient: shutdown 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f581408f0 msgr2=0x7f1f580acc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f581408f0 0x7f1f580acc80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000b990 tx=0x7f1f5000be50 comp rx=0 tx=0).stop 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 shutdown_connections 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f1f3c06c9a0 0x7f1f3c06ee60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f581408f0 0x7f1f580acc80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f5813d7a0 0x7f1f580aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 --2- 172.21.15.110:0/2532242959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f5806fc00 0x7f1f58135f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 >> 172.21.15.110:0/2532242959 conn(0x7f1f58066f00 msgr2=0x7f1f580b5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 shutdown_connections 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 -- 172.21.15.110:0/2532242959 wait complete. 2024-01-27T03:43:33.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.668+0000 7f1f5fe75640 1 librados: shutdown 2024-01-27T03:43:33.680 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.0 2024-01-27T03:43:33.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.707+0000 7f85d53d2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-27T03:43:33.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.707+0000 7f85d53d2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-27T03:43:33.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.707+0000 7f85d53d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:33.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.707+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f85d006edf0 con 0x7f85d00a98b0 2024-01-27T03:43:33.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.711+0000 7f85ccff9640 1 -- 172.21.15.110:0/3234982871 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f85d006edf0 con 0x7f85d00a98b0 2024-01-27T03:43:33.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.711+0000 7f85ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-27T03:43:33.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.711+0000 7f85ccff9640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:33.713 INFO:teuthology.orchestra.run.smithi110.stdout:68719476742 2024-01-27T03:43:33.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.713+0000 7f85d53d2640 10 librados: watch_flush enter 2024-01-27T03:43:33.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.713+0000 7f85d53d2640 10 librados: watch_flush exit 2024-01-27T03:43:33.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.713+0000 7f85d53d2640 20 client.4350.objecter shutdown clearing up homeless session... 2024-01-27T03:43:33.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.713+0000 7f85d53d2640 10 client.4350.objecter successfully canceled tick 2024-01-27T03:43:33.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f85bc06c9a0 msgr2=0x7f85bc06ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f85bc06c9a0 0x7f85bc06ee60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f85d01433c0 tx=0x7f85c4006c90 comp rx=0 tx=0).stop 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 10 monclient: shutdown 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85d00a98b0 msgr2=0x7f85d006df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85d00a98b0 0x7f85d006df00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f85c800ba50 tx=0x7f85c800bf10 comp rx=0 tx=0).stop 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 shutdown_connections 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f85bc06c9a0 0x7f85bc06ee60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85d00a9c90 0x7f85d0070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d00aaaf0 0x7f85d0070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 --2- 172.21.15.110:0/3234982871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85d00a98b0 0x7f85d006df00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 >> 172.21.15.110:0/3234982871 conn(0x7f85d00b3750 msgr2=0x7f85d00b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 shutdown_connections 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 -- 172.21.15.110:0/3234982871 wait complete. 2024-01-27T03:43:33.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.714+0000 7f85d53d2640 1 librados: shutdown 2024-01-27T03:43:33.726 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.3 2024-01-27T03:43:33.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.811+0000 7fb77c3c7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-27T03:43:33.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.811+0000 7fb77c3c7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-27T03:43:33.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.811+0000 7fb77c3c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:33.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.811+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb7740aceb0 con 0x7fb7740b4680 2024-01-27T03:43:33.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.815+0000 7fb76affd640 1 -- 172.21.15.110:0/1939885679 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb7740aceb0 con 0x7fb7740b4680 2024-01-27T03:43:33.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.815+0000 7fb76affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-27T03:43:33.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.815+0000 7fb76affd640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:33.816 INFO:teuthology.orchestra.run.smithi110.stdout:90194313224 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 10 librados: watch_flush enter 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 10 librados: watch_flush exit 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 20 client.4348.objecter shutdown clearing up homeless session... 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 10 client.4348.objecter successfully canceled tick 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 msgr2=0x7fb75806ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb770001560 tx=0x7fb770007990 comp rx=0 tx=0).stop 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 10 monclient: shutdown 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4680 msgr2=0x7fb7740aa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4680 0x7fb7740aa9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb764009870 tx=0x7fb7640046c0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 shutdown_connections 2024-01-27T03:43:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb75806c9a0 0x7fb75806ee60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7740ad4e0 0x7fb7740ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7740b4680 0x7fb7740aa9f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 --2- 172.21.15.110:0/1939885679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7740b3b30 0x7fb77413e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 >> 172.21.15.110:0/1939885679 conn(0x7fb774066490 msgr2=0x7fb7740b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 shutdown_connections 2024-01-27T03:43:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 -- 172.21.15.110:0/1939885679 wait complete. 2024-01-27T03:43:33.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.817+0000 7fb77c3c7640 1 librados: shutdown 2024-01-27T03:43:33.830 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.1 2024-01-27T03:43:33.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.863+0000 7f3028851640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-27T03:43:33.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.863+0000 7f3028851640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-27T03:43:33.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.863+0000 7f3028851640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:33.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.863+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f30240af760 con 0x7f30240b3b30 2024-01-27T03:43:33.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.906+0000 7f3012ffd640 1 -- 172.21.15.110:0/2018620571 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3014017090 con 0x7f30240b3b30 2024-01-27T03:43:33.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.906+0000 7f3012ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-27T03:43:33.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.906+0000 7f3012ffd640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:33.907 INFO:teuthology.orchestra.run.smithi110.stdout:90194313226 2024-01-27T03:43:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 10 librados: watch_flush enter 2024-01-27T03:43:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 10 librados: watch_flush exit 2024-01-27T03:43:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 20 client.4301.objecter shutdown clearing up homeless session... 2024-01-27T03:43:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 10 client.4301.objecter successfully canceled tick 2024-01-27T03:43:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f300406c9a0 msgr2=0x7f300406ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f300406c9a0 0x7f300406ee60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f30240aca10 tx=0x7f3018006c90 comp rx=0 tx=0).stop 2024-01-27T03:43:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 10 monclient: shutdown 2024-01-27T03:43:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:33.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b3b30 msgr2=0x7f30240ab350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b3b30 0x7f30240ab350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f301400c410 tx=0x7f301400c8d0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 shutdown_connections 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f300406c9a0 0x7f300406ee60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30240a94b0 0x7f30240a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30240b4680 0x7f302413aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 --2- 172.21.15.110:0/2018620571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240b3b30 0x7f30240ab350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 >> 172.21.15.110:0/2018620571 conn(0x7f3024066490 msgr2=0x7f30240b73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 shutdown_connections 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 -- 172.21.15.110:0/2018620571 wait complete. 2024-01-27T03:43:33.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:33.908+0000 7f3028851640 1 librados: shutdown 2024-01-27T03:43:33.920 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313226 for osd.2 2024-01-27T03:43:33.920 DEBUG:teuthology.parallel:result is None 2024-01-27T03:43:34.329 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.431+0000 7fc0a8234640 10 monclient: get_monmap_and_config 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.431+0000 7fc0a8234640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.431+0000 7fc0a8234640 10 monclient: monmap: 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:34.432169+0000 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:34.432169+0000 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:34.432 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:34.433 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:34.433 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:34.433 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.431+0000 7fc0a8234640 1 Processor -- start 2024-01-27T03:43:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.431+0000 7fc0a8234640 1 -- start start 2024-01-27T03:43:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.431+0000 7fc0a8234640 10 monclient: init 2024-01-27T03:43:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-01-27T03:43:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a00b4be0 0x7fc0a00b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 10 monclient(hunting): picked mon.noname-b con 0x7fc0a00b4be0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0a013bc80 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:34.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0a013c0e0 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a00aa220 con 0x7fc0a013bc80 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a013c570 con 0x7fc0a013c0e0 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a00b4fc0 con 0x7fc0a00b4be0 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a8234640 10 monclient(hunting): authenticate will time out at 1497.786389s 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a57cf640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc0a013bc80 0x7fc0a0145920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55460/0 (socket says 172.21.15.110:55460) 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.432+0000 7fc0a57cf640 1 -- 172.21.15.110:0/4084599985 learned_addr learned my addr 172.21.15.110:0/4084599985 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.433+0000 7fc0a5fd0640 1 --2- 172.21.15.110:0/4084599985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a00b4be0 0x7fc0a00b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.433+0000 7fc0a5fd0640 10 monclient(hunting): get_auth_request con 0x7fc0a00b4be0 auth_method 0 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.433+0000 7fc0a5fd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.433+0000 7fc0a5fd0640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.433+0000 7fc0a5fd0640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.433+0000 7fc0a5fd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.433+0000 7fc0a5fd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.433+0000 7fc0a5fd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 1 -- 172.21.15.110:0/4084599985 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013751705 0 0) 0x7fc098001110 con 0x7fc0a013c0e0 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 10 monclient(hunting): _negotiate 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 10 monclient(hunting): my global_id is 4351 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 1 -- 172.21.15.110:0/4084599985 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc08c003520 con 0x7fc0a013c0e0 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 1 -- 172.21.15.110:0/4084599985 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836721484 0 0) 0x7fc0a00aa220 con 0x7fc0a013bc80 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 10 monclient(hunting): _negotiate 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 10 monclient(hunting): my global_id is 4353 2024-01-27T03:43:34.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a4fce640 1 -- 172.21.15.110:0/4084599985 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc098001110 con 0x7fc0a013bc80 2024-01-27T03:43:34.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a5fd0640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-01-27T03:43:34.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a5fd0640 1 -- 172.21.15.110:0/4084599985 >> v1:172.21.15.162:6789/0 conn(0x7fc0a013c0e0 legacy=0x7fc0a0146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a5fd0640 1 -- 172.21.15.110:0/4084599985 >> v1:172.21.15.110:6789/0 conn(0x7fc0a013bc80 legacy=0x7fc0a0145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a5fd0640 10 monclient: _finish_hunting 0 2024-01-27T03:43:34.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a5fd0640 1 monclient: found mon.noname-b 2024-01-27T03:43:34.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a5fd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:34.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a5fd0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:34.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.435+0000 7fc0a5fd0640 1 -- 172.21.15.110:0/4084599985 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a00b43d0 con 0x7fc0a00b4be0 2024-01-27T03:43:34.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.437+0000 7fc0a5fd0640 1 --2- 172.21.15.110:0/4084599985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a00b4be0 0x7fc0a00b3e90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009dc0 tx=0x7fc09c02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12fef0aa72d925ec server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 1 -- 172.21.15.110:0/4084599985 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c02cbb0 con 0x7fc0a00b4be0 2024-01-27T03:43:34.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 10 monclient: dump: 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 10 monclient: _finish_auth 0 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 10 monclient: _check_auth_tickets 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 1 -- 172.21.15.110:0/4084599985 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c02cd10 con 0x7fc0a00b4be0 2024-01-27T03:43:34.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 5 monclient: authenticate success, global_id 4304 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 1 -- 172.21.15.110:0/4084599985 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c034660 con 0x7fc0a00b4be0 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a4fce640 10 monclient: dump: 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.443 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:34.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 4 set_mon_vals no callback set 2024-01-27T03:43:34.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 10 monclient: shutdown 2024-01-27T03:43:34.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:34.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 1 -- 172.21.15.110:0/4084599985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a00b4be0 msgr2=0x7fc0a00b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 1 --2- 172.21.15.110:0/4084599985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a00b4be0 0x7fc0a00b3e90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009dc0 tx=0x7fc09c02be00 comp rx=0 tx=0).stop 2024-01-27T03:43:34.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 1 -- 172.21.15.110:0/4084599985 shutdown_connections 2024-01-27T03:43:34.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 1 --2- 172.21.15.110:0/4084599985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a00b4be0 0x7fc0a00b3e90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 1 -- 172.21.15.110:0/4084599985 >> 172.21.15.110:0/4084599985 conn(0x7fc0a0066490 msgr2=0x7fc0a00668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:34.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 1 -- 172.21.15.110:0/4084599985 shutdown_connections 2024-01-27T03:43:34.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.441+0000 7fc0a8234640 1 -- 172.21.15.110:0/4084599985 wait complete. 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.442+0000 7fc0a8234640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.442+0000 7fc0a8234640 10 monclient: monmap: 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.442+0000 7fc0a8234640 1 librados: starting msgr at 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.442+0000 7fc0a8234640 1 librados: starting objecter 2024-01-27T03:43:34.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.442+0000 7fc0a8234640 1 Processor -- start 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.442+0000 7fc0a8234640 1 -- start start 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.442+0000 7fc0a8234640 1 librados: setting wanted keys 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.442+0000 7fc0a8234640 1 librados: calling monclient init 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.442+0000 7fc0a8234640 10 monclient: init 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0a00b4be0 0x7fc0a0142070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient(hunting): picked mon.noname-b con 0x7fc0a00b4be0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a013bc80 0x7fc0a00ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient(hunting): picked mon.noname-c con 0x7fc0a013bc80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0a013c0e0 0x7fc0a0139220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient(hunting): picked mon.noname-a con 0x7fc0a013c0e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a0139f60 con 0x7fc0a013c0e0 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a00abc30 con 0x7fc0a00b4be0 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a013a190 con 0x7fc0a013bc80 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a8234640 10 monclient(hunting): authenticate will time out at 1497.797319s 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a67d1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0a013c0e0 0x7fc0a0139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.443+0000 7fc0a5fd0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0a00b4be0 0x7fc0a0142070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a57cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a013bc80 0x7fc0a00ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a67d1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0a013c0e0 0x7fc0a0139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50208/0 (socket says 172.21.15.110:50208) 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a67d1640 1 -- 172.21.15.110:0/413481326 learned_addr learned my addr 172.21.15.110:0/413481326 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a67d1640 10 monclient(hunting): get_auth_request con 0x7fc0a013c0e0 auth_method 0 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a67d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a67d1640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a67d1640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a5fd0640 10 monclient(hunting): get_auth_request con 0x7fc0a00b4be0 auth_method 0 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a5fd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a5fd0640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a5fd0640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a57cf640 10 monclient(hunting): get_auth_request con 0x7fc0a013bc80 auth_method 0 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a57cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a57cf640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a57cf640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a5fd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a5fd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a5fd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a67d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a67d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a67d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a57cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a57cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.444+0000 7fc0a57cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 1 -- 172.21.15.110:0/413481326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a013bc80 msgr2=0x7fc0a00ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a013bc80 0x7fc0a00ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 1 -- 172.21.15.110:0/413481326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0a013c0e0 msgr2=0x7fc0a0139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0a013c0e0 0x7fc0a0139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 10 monclient: _finish_hunting 0 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 1 monclient: found mon.noname-b 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.447+0000 7fc0a5fd0640 1 -- 172.21.15.110:0/413481326 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c009a70 con 0x7fc0a00b4be0 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.449+0000 7fc0a5fd0640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0a00b4be0 0x7fc0a0142070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc09c033000 tx=0x7fc09c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.449+0000 7fc0967fc640 10 client.?.objecter ms_handle_connect 0x7fc0a00b4be0 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.449+0000 7fc0967fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0967fc640 1 -- 172.21.15.110:0/413481326 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c03e610 con 0x7fc0a00b4be0 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0967fc640 10 monclient: dump: 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.454 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0967fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0967fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0a8234640 5 monclient: authenticate success, global_id 4354 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0a8234640 10 monclient: _renew_subs 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0a8234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.451+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a013a4a0 con 0x7fc0a00b4be0 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.452+0000 7fc0a8234640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.452+0000 7fc0a8234640 10 monclient: _renew_subs 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.452+0000 7fc0a8234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.452+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a0139760 con 0x7fc0a00b4be0 2024-01-27T03:43:34.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.452+0000 7fc0a8234640 1 librados: init done 2024-01-27T03:43:34.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.452+0000 7fc0a8234640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.452+0000 7fc0a8234640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.452+0000 7fc0a8234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:34.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.452+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a004ea50 con 0x7fc0a00b4be0 2024-01-27T03:43:34.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-27T03:43:34.548 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-27T03:43:34.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.600+0000 7f7b3f55e640 10 monclient: get_monmap_and_config 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.600+0000 7f7b3f55e640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.600+0000 7f7b3f55e640 10 monclient: monmap: 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:34.601639+0000 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:34.601639+0000 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:34.602 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:34.603 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:34.603 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:34.603 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:34.603 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.601+0000 7f7b3f55e640 1 Processor -- start 2024-01-27T03:43:34.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.601+0000 7f7b3f55e640 1 -- start start 2024-01-27T03:43:34.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.601+0000 7f7b3f55e640 10 monclient: init 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.601+0000 7f7b3f55e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.601+0000 7f7b3f55e640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.601+0000 7f7b3f55e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b380b4680 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.601+0000 7f7b3f55e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b4a60 0x7f7b380b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.601+0000 7f7b3f55e640 10 monclient(hunting): picked mon.noname-b con 0x7f7b380b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3f55e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b380b3b30 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3f55e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b380afbe0 con 0x7f7b380b4680 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3f55e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b380b0890 con 0x7f7b380b3b30 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3f55e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3f55e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b38142240 con 0x7f7b380b4a60 2024-01-27T03:43:34.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3f55e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3f55e640 10 monclient(hunting): authenticate will time out at 1497.956009s 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3d2fa640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7b380b4680 0x7f7b38142970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55464/0 (socket says 172.21.15.110:55464) 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3d2fa640 1 -- 172.21.15.110:0/2281479376 learned_addr learned my addr 172.21.15.110:0/2281479376 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.602+0000 7f7b3caf9640 1 --2- 172.21.15.110:0/2281479376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b4a60 0x7f7b380b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.603+0000 7f7b3caf9640 10 monclient(hunting): get_auth_request con 0x7f7b380b4a60 auth_method 0 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.603+0000 7f7b3caf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.603+0000 7f7b3caf9640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.603+0000 7f7b3caf9640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.603+0000 7f7b3caf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.603+0000 7f7b3caf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.603+0000 7f7b3caf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.605+0000 7f7b2bfff640 1 -- 172.21.15.110:0/2281479376 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416803289 0 0) 0x7f7b380afbe0 con 0x7f7b380b4680 2024-01-27T03:43:34.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.605+0000 7f7b2bfff640 10 monclient(hunting): _negotiate 2024-01-27T03:43:34.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.605+0000 7f7b2bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.605+0000 7f7b2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.605+0000 7f7b2bfff640 10 monclient(hunting): my global_id is 4359 2024-01-27T03:43:34.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.605+0000 7f7b2bfff640 1 -- 172.21.15.110:0/2281479376 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b24003520 con 0x7f7b380b4680 2024-01-27T03:43:34.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.606+0000 7f7b3caf9640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-01-27T03:43:34.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.606+0000 7f7b3caf9640 1 -- 172.21.15.110:0/2281479376 >> v1:172.21.15.162:6789/0 conn(0x7f7b380b3b30 legacy=0x7f7b380b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:34.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.606+0000 7f7b3caf9640 1 -- 172.21.15.110:0/2281479376 >> v1:172.21.15.110:6789/0 conn(0x7f7b380b4680 legacy=0x7f7b38142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.606+0000 7f7b3caf9640 10 monclient: _finish_hunting 0 2024-01-27T03:43:34.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.606+0000 7f7b3caf9640 1 monclient: found mon.noname-b 2024-01-27T03:43:34.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.606+0000 7f7b3caf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:34.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.606+0000 7f7b3caf9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:34.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.606+0000 7f7b3caf9640 1 -- 172.21.15.110:0/2281479376 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b380b0b00 con 0x7f7b380b4a60 2024-01-27T03:43:34.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.608+0000 7f7b3caf9640 1 --2- 172.21.15.110:0/2281479376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b4a60 0x7f7b380b1470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c0097d0 tx=0x7f7b2c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa93f24823c2f5a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.612 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-27T03:43:34.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 1 -- 172.21.15.110:0/2281479376 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c031770 con 0x7f7b380b4a60 2024-01-27T03:43:34.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:34.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 10 monclient: dump: 2024-01-27T03:43:34.616 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.616 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 10 monclient: _finish_auth 0 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 10 monclient: _check_auth_tickets 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 1 -- 172.21.15.110:0/2281479376 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c0318d0 con 0x7f7b380b4a60 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b3f55e640 5 monclient: authenticate success, global_id 4310 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 1 -- 172.21.15.110:0/2281479376 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c031c60 con 0x7f7b380b4a60 2024-01-27T03:43:34.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b3f55e640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b3f55e640 4 set_mon_vals no callback set 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.613+0000 7f7b2bfff640 10 monclient: dump: 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.614+0000 7f7b3f55e640 10 monclient: shutdown 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.614+0000 7f7b3f55e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.614+0000 7f7b3f55e640 1 -- 172.21.15.110:0/2281479376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b4a60 msgr2=0x7f7b380b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.614+0000 7f7b3f55e640 1 --2- 172.21.15.110:0/2281479376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b4a60 0x7f7b380b1470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c0097d0 tx=0x7f7b2c02b810 comp rx=0 tx=0).stop 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.614+0000 7f7b3f55e640 1 -- 172.21.15.110:0/2281479376 shutdown_connections 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.614+0000 7f7b3f55e640 1 --2- 172.21.15.110:0/2281479376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b4a60 0x7f7b380b1470 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.614+0000 7f7b3f55e640 1 -- 172.21.15.110:0/2281479376 >> 172.21.15.110:0/2281479376 conn(0x7f7b38066490 msgr2=0x7f7b380668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.614+0000 7f7b3f55e640 1 -- 172.21.15.110:0/2281479376 shutdown_connections 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.614+0000 7f7b3f55e640 1 -- 172.21.15.110:0/2281479376 wait complete. 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.615+0000 7f7b3f55e640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.615+0000 7f7b3f55e640 10 monclient: monmap: 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:34.619 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.615+0000 7f7b3f55e640 1 librados: starting msgr at 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.615+0000 7f7b3f55e640 1 librados: starting objecter 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.615+0000 7f7b3f55e640 1 Processor -- start 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.615+0000 7f7b3f55e640 1 -- start start 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.615+0000 7f7b3f55e640 1 librados: setting wanted keys 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.615+0000 7f7b3f55e640 1 librados: calling monclient init 2024-01-27T03:43:34.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.615+0000 7f7b3f55e640 10 monclient: init 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b3b30 0x7f7b381d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient(hunting): picked mon.noname-c con 0x7f7b380b3b30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b380b4680 0x7f7b380ad640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient(hunting): picked mon.noname-a con 0x7f7b380b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b381389f0 0x7f7b38138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient(hunting): picked mon.noname-b con 0x7f7b381389f0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b381435d0 con 0x7f7b380b4680 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b380adb80 con 0x7f7b381389f0 2024-01-27T03:43:34.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b38143840 con 0x7f7b380b3b30 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3f55e640 10 monclient(hunting): authenticate will time out at 1497.970245s 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3caf9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b380b4680 0x7f7b380ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.616+0000 7f7b3d2fa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b3b30 0x7f7b381d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3caf9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b380b4680 0x7f7b380ad640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50214/0 (socket says 172.21.15.110:50214) 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3caf9640 1 -- 172.21.15.110:0/1082160130 learned_addr learned my addr 172.21.15.110:0/1082160130 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3caf9640 10 monclient(hunting): get_auth_request con 0x7f7b380b4680 auth_method 0 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3caf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3caf9640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3caf9640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3d2fa640 10 monclient(hunting): get_auth_request con 0x7f7b380b3b30 auth_method 0 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3d2fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3d2fa640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3d2fa640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3caf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3caf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3caf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3d2fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3d2fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.617+0000 7f7b3d2fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.621+0000 7f7b3d2fa640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3d2fa640 1 -- 172.21.15.110:0/1082160130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b381389f0 msgr2=0x7f7b38138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3d2fa640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b381389f0 0x7f7b38138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3d2fa640 1 -- 172.21.15.110:0/1082160130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b380b4680 msgr2=0x7f7b380ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3d2fa640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b380b4680 0x7f7b380ad640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3d2fa640 10 monclient: _finish_hunting 0 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3d2fa640 1 monclient: found mon.noname-c 2024-01-27T03:43:34.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3d2fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:34.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3d2fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:34.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3d2fa640 1 -- 172.21.15.110:0/1082160130 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c009400 con 0x7f7b380b3b30 2024-01-27T03:43:34.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.622+0000 7f7b3caf9640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b380b4680 0x7f7b380ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T03:43:34.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.625+0000 7f7b3d2fa640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b3b30 0x7f7b381d2c30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400c430 tx=0x7f7b3400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.625+0000 7f7b29ffb640 10 client.?.objecter ms_handle_connect 0x7f7b380b3b30 2024-01-27T03:43:34.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.625+0000 7f7b29ffb640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.628+0000 7f7b29ffb640 1 -- 172.21.15.110:0/1082160130 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34005ce0 con 0x7f7b380b3b30 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.628+0000 7f7b29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.628+0000 7f7b29ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.628+0000 7f7b29ffb640 10 monclient: dump: 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.630 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.628+0000 7f7b29ffb640 10 monclient: _finish_auth 0 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.628+0000 7f7b29ffb640 10 monclient: _check_auth_tickets 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.628+0000 7f7b29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.628+0000 7f7b29ffb640 1 -- 172.21.15.110:0/1082160130 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b34005e40 con 0x7f7b380b3b30 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.628+0000 7f7b3f55e640 5 monclient: authenticate success, global_id 4313 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b3f55e640 10 monclient: _renew_subs 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b3f55e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b38143b20 con 0x7f7b380b3b30 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b29ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b29ffb640 1 -- 172.21.15.110:0/1082160130 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b340072f0 con 0x7f7b380b3b30 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b29ffb640 10 monclient: dump: 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.631 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b2b7fe640 4 set_mon_vals no callback set 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b3f55e640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:34.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b3f55e640 10 monclient: _renew_subs 2024-01-27T03:43:34.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b3f55e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:34.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b380ac9a0 con 0x7f7b380b3b30 2024-01-27T03:43:34.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.629+0000 7f7b3f55e640 1 librados: init done 2024-01-27T03:43:34.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.630+0000 7f7b3f55e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.630+0000 7f7b3f55e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.630+0000 7f7b3f55e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:34.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.630+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b38074d10 con 0x7f7b380b3b30 2024-01-27T03:43:34.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.649+0000 7f7b29ffb640 1 -- 172.21.15.110:0/1082160130 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f7b34007450 con 0x7f7b380b3b30 2024-01-27T03:43:34.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.650+0000 7f7b29ffb640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7b1c06c9d0 0x7f7b1c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.650+0000 7f7b3caf9640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7b1c06c9d0 0x7f7b1c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.650+0000 7f7b3caf9640 10 monclient: get_auth_request con 0x7f7b1c06c9d0 auth_method 0 2024-01-27T03:43:34.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.651+0000 7f7b29ffb640 10 client.4313.objecter ms_handle_connect 0x7f7b1c06c9d0 2024-01-27T03:43:34.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.651+0000 7f7b3caf9640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7b1c06c9d0 0x7f7b1c06ee90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c000c00 tx=0x7f7b2c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.653+0000 7f7b29ffb640 1 -- 172.21.15.110:0/1082160130 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7b34007e20 con 0x7f7b380b3b30 2024-01-27T03:43:34.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.653+0000 7f7b29ffb640 10 client.4313.objecter ms_dispatch 0x7f7b38000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:34.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.653+0000 7f7b29ffb640 3 client.4313.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:34.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.653+0000 7f7b29ffb640 3 client.4313.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:34.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.653+0000 7f7b29ffb640 20 client.4313.objecter dump_active .. 0 homeless 2024-01-27T03:43:34.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.656+0000 7ff158705640 10 monclient: get_monmap_and_config 2024-01-27T03:43:34.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.656+0000 7ff158705640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.656+0000 7ff158705640 10 monclient: monmap: 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:34.658066+0000 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:34.658066+0000 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:34.658 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.657+0000 7ff158705640 1 Processor -- start 2024-01-27T03:43:34.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.657+0000 7ff158705640 1 -- start start 2024-01-27T03:43:34.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.657+0000 7ff158705640 10 monclient: init 2024-01-27T03:43:34.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-01-27T03:43:34.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff15006fc00 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:34.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff1501403b0 0x7ff1501407f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 10 monclient(hunting): picked mon.noname-a con 0x7ff1501403b0 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1501487d0 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1500b0460 con 0x7ff15006fc00 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff150071b70 con 0x7ff1501487d0 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff150071e50 con 0x7ff1501403b0 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.658+0000 7ff158705640 10 monclient(hunting): authenticate will time out at 1498.012352s 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.659+0000 7ff155ca0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff1501403b0 0x7ff1501407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.659+0000 7ff1564a1640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff15006fc00 0x7ff15013d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55478/0 (socket says 172.21.15.110:55478) 2024-01-27T03:43:34.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.659+0000 7ff1564a1640 1 -- 172.21.15.110:0/1764664131 learned_addr learned my addr 172.21.15.110:0/1764664131 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:34.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.660+0000 7ff155ca0640 10 monclient(hunting): get_auth_request con 0x7ff1501403b0 auth_method 0 2024-01-27T03:43:34.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.660+0000 7ff155ca0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.660+0000 7ff155ca0640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.660+0000 7ff155ca0640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.662+0000 7ff155ca0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.662+0000 7ff155ca0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.662+0000 7ff155ca0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.662+0000 7ff15549f640 1 -- 172.21.15.110:0/1764664131 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066862222 0 0) 0x7ff1500b0460 con 0x7ff15006fc00 2024-01-27T03:43:34.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.662+0000 7ff15549f640 10 monclient(hunting): _negotiate 2024-01-27T03:43:34.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.662+0000 7ff15549f640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.662+0000 7ff15549f640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.662+0000 7ff15549f640 10 monclient(hunting): my global_id is 4365 2024-01-27T03:43:34.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.662+0000 7ff15549f640 1 -- 172.21.15.110:0/1764664131 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c003520 con 0x7ff15006fc00 2024-01-27T03:43:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.667+0000 7ff155ca0640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-01-27T03:43:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.667+0000 7ff155ca0640 1 -- 172.21.15.110:0/1764664131 >> v1:172.21.15.162:6789/0 conn(0x7ff1501487d0 legacy=0x7ff15014abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.667+0000 7ff155ca0640 1 -- 172.21.15.110:0/1764664131 >> v1:172.21.15.110:6789/0 conn(0x7ff15006fc00 legacy=0x7ff15013d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.667+0000 7ff155ca0640 10 monclient: _finish_hunting 0 2024-01-27T03:43:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.667+0000 7ff155ca0640 1 monclient: found mon.noname-a 2024-01-27T03:43:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.667+0000 7ff155ca0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.667+0000 7ff155ca0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.667+0000 7ff155ca0640 1 -- 172.21.15.110:0/1764664131 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff150072130 con 0x7ff1501403b0 2024-01-27T03:43:34.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.671+0000 7ff155ca0640 1 --2- 172.21.15.110:0/1764664131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501403b0 0x7ff1501407f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff1400097a0 tx=0x7ff14002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6cdf2b9dac07288 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 1 -- 172.21.15.110:0/1764664131 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140031830 con 0x7ff1501403b0 2024-01-27T03:43:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 10 monclient: dump: 2024-01-27T03:43:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 10 monclient: _finish_auth 0 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 10 monclient: _check_auth_tickets 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 1 -- 172.21.15.110:0/1764664131 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff140031990 con 0x7ff1501403b0 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 1 -- 172.21.15.110:0/1764664131 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140031d20 con 0x7ff1501403b0 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff158705640 5 monclient: authenticate success, global_id 4368 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff158705640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff158705640 4 set_mon_vals no callback set 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.676+0000 7ff15549f640 10 monclient: dump: 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.679 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.677+0000 7ff158705640 10 monclient: shutdown 2024-01-27T03:43:34.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.677+0000 7ff158705640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:34.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.677+0000 7ff158705640 1 -- 172.21.15.110:0/1764664131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501403b0 msgr2=0x7ff1501407f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.677+0000 7ff158705640 1 --2- 172.21.15.110:0/1764664131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501403b0 0x7ff1501407f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff1400097a0 tx=0x7ff14002b8d0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.677+0000 7ff158705640 1 -- 172.21.15.110:0/1764664131 shutdown_connections 2024-01-27T03:43:34.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.677+0000 7ff158705640 1 --2- 172.21.15.110:0/1764664131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501403b0 0x7ff1501407f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.677+0000 7ff158705640 1 -- 172.21.15.110:0/1764664131 >> 172.21.15.110:0/1764664131 conn(0x7ff150066f00 msgr2=0x7ff1500b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:34.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.677+0000 7ff158705640 1 -- 172.21.15.110:0/1764664131 shutdown_connections 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.677+0000 7ff158705640 1 -- 172.21.15.110:0/1764664131 wait complete. 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.678+0000 7ff158705640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.678+0000 7ff158705640 10 monclient: monmap: 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.678+0000 7ff158705640 1 librados: starting msgr at 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.678+0000 7ff158705640 1 librados: starting objecter 2024-01-27T03:43:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.678+0000 7ff158705640 1 Processor -- start 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.678+0000 7ff158705640 1 -- start start 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.678+0000 7ff158705640 1 librados: setting wanted keys 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.678+0000 7ff158705640 1 librados: calling monclient init 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.678+0000 7ff158705640 10 monclient: init 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff15006fc00 0x7ff15006d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient(hunting): picked mon.noname-b con 0x7ff15006fc00 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1501403b0 0x7ff150134c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient(hunting): picked mon.noname-c con 0x7ff1501403b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501487d0 0x7ff150135140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient(hunting): picked mon.noname-a con 0x7ff1501487d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1500b3c00 con 0x7ff1501487d0 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff150135680 con 0x7ff15006fc00 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15013c390 con 0x7ff1501403b0 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff158705640 10 monclient(hunting): authenticate will time out at 1498.033438s 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.679+0000 7ff156ca2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501487d0 0x7ff150135140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.680+0000 7ff156ca2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501487d0 0x7ff150135140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50236/0 (socket says 172.21.15.110:50236) 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.680+0000 7ff156ca2640 1 -- 172.21.15.110:0/2078080876 learned_addr learned my addr 172.21.15.110:0/2078080876 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.680+0000 7ff156ca2640 10 monclient(hunting): get_auth_request con 0x7ff1501487d0 auth_method 0 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.680+0000 7ff156ca2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.680+0000 7ff156ca2640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.680+0000 7ff156ca2640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.685 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-27T03:43:34.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.680+0000 7ff156ca2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.680+0000 7ff156ca2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.680+0000 7ff156ca2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.683+0000 7ff156ca2640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2024-01-27T03:43:34.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.683+0000 7ff156ca2640 1 -- 172.21.15.110:0/2078080876 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1501403b0 msgr2=0x7ff150134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.683+0000 7ff156ca2640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1501403b0 0x7ff150134c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.683+0000 7ff156ca2640 1 -- 172.21.15.110:0/2078080876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff15006fc00 msgr2=0x7ff15006d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.684+0000 7ff156ca2640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff15006fc00 0x7ff15006d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.684+0000 7ff156ca2640 10 monclient: _finish_hunting 0 2024-01-27T03:43:34.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.684+0000 7ff156ca2640 1 monclient: found mon.noname-a 2024-01-27T03:43:34.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.684+0000 7ff156ca2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:34.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.684+0000 7ff156ca2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:34.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.684+0000 7ff156ca2640 1 -- 172.21.15.110:0/2078080876 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff140009400 con 0x7ff1501487d0 2024-01-27T03:43:34.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.686+0000 7ff156ca2640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501487d0 0x7ff150135140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff14800b9e0 tx=0x7ff14800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.686+0000 7ff146ffd640 10 client.?.objecter ms_handle_connect 0x7ff1501487d0 2024-01-27T03:43:34.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.686+0000 7ff146ffd640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff146ffd640 1 -- 172.21.15.110:0/2078080876 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14800cb20 con 0x7ff1501487d0 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff146ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff146ffd640 10 monclient: dump: 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.694 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff146ffd640 10 monclient: _finish_auth 0 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff146ffd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff146ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff146ffd640 1 -- 172.21.15.110:0/2078080876 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14800cc80 con 0x7ff1501487d0 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff158705640 5 monclient: authenticate success, global_id 4371 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff146ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.690+0000 7ff154c9e640 4 set_mon_vals no callback set 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff146ffd640 1 -- 172.21.15.110:0/2078080876 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148014680 con 0x7ff1501487d0 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff146ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff146ffd640 10 monclient: dump: 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.695 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff158705640 10 monclient: _renew_subs 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff158705640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff150140eb0 con 0x7ff1501487d0 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff158705640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff158705640 10 monclient: _renew_subs 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff158705640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1500b3890 con 0x7ff1501487d0 2024-01-27T03:43:34.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.691+0000 7ff158705640 1 librados: init done 2024-01-27T03:43:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.693+0000 7ff158705640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.693+0000 7ff158705640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.693+0000 7ff158705640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:34.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.693+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff150074d10 con 0x7ff1501487d0 2024-01-27T03:43:34.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.699+0000 7fc0967fc640 1 -- 172.21.15.110:0/413481326 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c03e770 con 0x7fc0a00b4be0 2024-01-27T03:43:34.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.699+0000 7fc0967fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.699+0000 7fc0967fc640 1 -- 172.21.15.110:0/413481326 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c03eb20 con 0x7fc0a00b4be0 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.699+0000 7fc0967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.699+0000 7fc097fff640 4 set_mon_vals no callback set 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.699+0000 7fc0967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.699+0000 7fc0967fc640 10 monclient: dump: 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.701 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.702 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.700+0000 7fc0967fc640 1 -- 172.21.15.110:0/413481326 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fc09c03b050 con 0x7fc0a00b4be0 2024-01-27T03:43:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.701+0000 7fc0967fc640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fc08406c9a0 0x7fc08406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.701+0000 7fc0a57cf640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fc08406c9a0 0x7fc08406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.701+0000 7fc0a57cf640 10 monclient: get_auth_request con 0x7fc08406c9a0 auth_method 0 2024-01-27T03:43:34.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.702+0000 7fc0a57cf640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fc08406c9a0 0x7fc08406ee60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0136b30 tx=0x7fc090009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.702+0000 7fc0967fc640 10 client.4354.objecter ms_handle_connect 0x7fc08406c9a0 2024-01-27T03:43:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.705+0000 7fc0967fc640 1 -- 172.21.15.110:0/413481326 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc09c03ee30 con 0x7fc0a00b4be0 2024-01-27T03:43:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.705+0000 7fc0967fc640 10 client.4354.objecter ms_dispatch 0x7fc0a0000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.705+0000 7fc0967fc640 3 client.4354.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.705+0000 7fc0967fc640 3 client.4354.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:34.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.705+0000 7fc0967fc640 20 client.4354.objecter dump_active .. 0 homeless 2024-01-27T03:43:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.717+0000 7fc0967fc640 1 -- 172.21.15.110:0/413481326 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fc09c0772d0 con 0x7fc0a00b4be0 2024-01-27T03:43:34.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.717+0000 7fc0967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.717+0000 7fc0967fc640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.718+0000 7ff146ffd640 1 -- 172.21.15.110:0/2078080876 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7ff14801d3b0 con 0x7ff1501487d0 2024-01-27T03:43:34.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.719+0000 7ff146ffd640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff13406c9a0 0x7ff13406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.719+0000 7ff1564a1640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff13406c9a0 0x7ff13406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.719+0000 7ff1564a1640 10 monclient: get_auth_request con 0x7ff13406c9a0 auth_method 0 2024-01-27T03:43:34.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.719+0000 7ff1564a1640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff13406c9a0 0x7ff13406ee60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff14c000f60 tx=0x7ff14c0078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.719+0000 7ff146ffd640 10 client.4371.objecter ms_handle_connect 0x7ff13406c9a0 2024-01-27T03:43:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.721+0000 7ff146ffd640 1 -- 172.21.15.110:0/2078080876 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff14800ce30 con 0x7ff1501487d0 2024-01-27T03:43:34.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.721+0000 7ff146ffd640 10 client.4371.objecter ms_dispatch 0x7ff150000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.721+0000 7ff146ffd640 3 client.4371.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.721+0000 7ff146ffd640 3 client.4371.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:34.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.721+0000 7ff146ffd640 20 client.4371.objecter dump_active .. 0 homeless 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.725+0000 7fb3d6ba9640 10 monclient: get_monmap_and_config 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.725+0000 7fb3d6ba9640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.725+0000 7fb3d6ba9640 10 monclient: monmap: 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:34.726256+0000 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:34.726256+0000 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:34.726 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:34.727 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:34.727 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.727 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-27T03:43:34.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.725+0000 7fb3d6ba9640 1 Processor -- start 2024-01-27T03:43:34.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.725+0000 7fb3d6ba9640 1 -- start start 2024-01-27T03:43:34.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.725+0000 7fb3d6ba9640 10 monclient: init 2024-01-27T03:43:34.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-01-27T03:43:34.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3d00b4680 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:34.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4a60 0x7fb3d00b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 10 monclient(hunting): picked mon.noname-b con 0x7fb3d00b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb3d00b3b30 0x7fb3d00b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 10 monclient(hunting): picked mon.noname-a con 0x7fb3d00b3b30 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00afbe0 con 0x7fb3d00b4680 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d0142240 con 0x7fb3d00b3b30 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d00b0890 con 0x7fb3d00b4a60 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.726+0000 7fb3d6ba9640 10 monclient(hunting): authenticate will time out at 1498.080844s 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.727+0000 7fb3d4945640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb3d00b4680 0x7fb3d0142970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:49202/0 (socket says 172.21.15.110:49202) 2024-01-27T03:43:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.727+0000 7fb3d4945640 1 -- 172.21.15.110:0/1759675993 learned_addr learned my addr 172.21.15.110:0/1759675993 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:34.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.794+0000 7f6292d33640 10 monclient: get_monmap_and_config 2024-01-27T03:43:34.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.794+0000 7f6292d33640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.794+0000 7f6292d33640 10 monclient: monmap: 2024-01-27T03:43:34.795 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.795 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.795 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:34.795323+0000 2024-01-27T03:43:34.795 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:34.795323+0000 2024-01-27T03:43:34.796 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.796 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.796 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:34.796 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:34.796 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:34.796 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:34.796 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:34.796 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.794+0000 7f6292d33640 1 Processor -- start 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.794+0000 7f6292d33640 1 -- start start 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 10 monclient: init 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f628c0b4680 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f628c0b4a60 0x7f628c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 10 monclient(hunting): picked mon.noname-c con 0x7f628c0b4a60 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f628c0b3b30 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0afbe0 con 0x7f628c0b3b30 2024-01-27T03:43:34.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0b0890 con 0x7f628c0b4680 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628c142240 con 0x7f628c0b4a60 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.795+0000 7f6292d33640 10 monclient(hunting): authenticate will time out at 1498.149809s 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.796+0000 7f6290acf640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f628c0b4680 0x7f628c142970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:49214/0 (socket says 172.21.15.110:49214) 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.796+0000 7f6290acf640 1 -- 172.21.15.110:0/3650285184 learned_addr learned my addr 172.21.15.110:0/3650285184 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.796+0000 7f628bfff640 1 --2- 172.21.15.110:0/3650285184 >> v2:172.21.15.162:3300/0 conn(0x7f628c0b4a60 0x7f628c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.797+0000 7f628bfff640 10 monclient(hunting): get_auth_request con 0x7f628c0b4a60 auth_method 0 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.797+0000 7f628bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.797+0000 7f628bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.797+0000 7f628bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.797+0000 7f628bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.797+0000 7f628bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.797+0000 7f628bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.800+0000 7f628bfff640 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2024-01-27T03:43:34.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.800+0000 7f628bfff640 1 -- 172.21.15.110:0/3650285184 >> v1:172.21.15.162:6789/0 conn(0x7f628c0b4680 legacy=0x7f628c142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.800+0000 7f628bfff640 1 -- 172.21.15.110:0/3650285184 >> v1:172.21.15.110:6789/0 conn(0x7f628c0b3b30 legacy=0x7f628c0b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:34.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.800+0000 7f628bfff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:34.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.800+0000 7f628bfff640 1 monclient: found mon.noname-c 2024-01-27T03:43:34.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.800+0000 7f628bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:34.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.800+0000 7f628bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:34.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.800+0000 7f628bfff640 1 -- 172.21.15.110:0/3650285184 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c0b0b00 con 0x7f628c0b4a60 2024-01-27T03:43:34.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.803+0000 7f628bfff640 1 --2- 172.21.15.110:0/3650285184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4a60 0x7f628c0b1470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f62780097a0 tx=0x7f627802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c17c1c45705ad1e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 1 -- 172.21.15.110:0/3650285184 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627802e070 con 0x7f628c0b4a60 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 10 monclient: dump: 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 10 monclient: _finish_auth 0 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 10 monclient: _check_auth_tickets 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 1 -- 172.21.15.110:0/3650285184 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62780315a0 con 0x7f628c0b4a60 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f6292d33640 5 monclient: authenticate success, global_id 4357 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f6292d33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f6292d33640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 1 -- 172.21.15.110:0/3650285184 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278031930 con 0x7f628c0b4a60 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f6292d33640 4 set_mon_vals no callback set 2024-01-27T03:43:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f628b7fe640 10 monclient: dump: 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.810 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f6292d33640 10 monclient: shutdown 2024-01-27T03:43:34.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f6292d33640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:34.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f6292d33640 1 -- 172.21.15.110:0/3650285184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4a60 msgr2=0x7f628c0b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.807+0000 7f6292d33640 1 --2- 172.21.15.110:0/3650285184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4a60 0x7f628c0b1470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f62780097a0 tx=0x7f627802b840 comp rx=0 tx=0).stop 2024-01-27T03:43:34.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.808+0000 7f6292d33640 1 -- 172.21.15.110:0/3650285184 shutdown_connections 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.808+0000 7f6292d33640 1 --2- 172.21.15.110:0/3650285184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4a60 0x7f628c0b1470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.808+0000 7f6292d33640 1 -- 172.21.15.110:0/3650285184 >> 172.21.15.110:0/3650285184 conn(0x7f628c066490 msgr2=0x7f628c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.808+0000 7f6292d33640 1 -- 172.21.15.110:0/3650285184 shutdown_connections 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.808+0000 7f6292d33640 1 -- 172.21.15.110:0/3650285184 wait complete. 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.809+0000 7f6292d33640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.809+0000 7f6292d33640 10 monclient: monmap: 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.815 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.809+0000 7f6292d33640 1 librados: starting msgr at 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.809+0000 7f6292d33640 1 librados: starting objecter 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.809+0000 7f6292d33640 1 Processor -- start 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.809+0000 7f6292d33640 1 -- start start 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.809+0000 7f6292d33640 1 librados: setting wanted keys 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.809+0000 7f6292d33640 1 librados: calling monclient init 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.809+0000 7f6292d33640 10 monclient: init 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f628c0b3b30 0x7f628c14ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient(hunting): picked mon.noname-a con 0x7f628c0b3b30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4680 0x7f628c138d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient(hunting): picked mon.noname-b con 0x7f628c0b4680 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f628c139240 0x7f628c0a9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient(hunting): picked mon.noname-c con 0x7f628c139240 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0aacc0 con 0x7f628c0b3b30 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c143560 con 0x7f628c0b4680 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c143840 con 0x7f628c139240 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.810+0000 7f6292d33640 10 monclient(hunting): authenticate will time out at 1498.164942s 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4680 0x7f628c138d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4680 0x7f628c138d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40830/0 (socket says 172.21.15.110:40830) 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 1 -- 172.21.15.110:0/2632807052 learned_addr learned my addr 172.21.15.110:0/2632807052 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 10 monclient(hunting): get_auth_request con 0x7f628c0b4680 auth_method 0 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:34.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.812+0000 7f628bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.814+0000 7f628bfff640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.815+0000 7f628bfff640 1 -- 172.21.15.110:0/2632807052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f628c139240 msgr2=0x7f628c0a9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.815+0000 7f628bfff640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f628c139240 0x7f628c0a9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.815+0000 7f628bfff640 1 -- 172.21.15.110:0/2632807052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f628c0b3b30 msgr2=0x7f628c14ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.815+0000 7f628bfff640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f628c0b3b30 0x7f628c14ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.815+0000 7f628bfff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.815+0000 7f628bfff640 1 monclient: found mon.noname-b 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.815+0000 7f628bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.815+0000 7f628bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:34.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.815+0000 7f628bfff640 1 -- 172.21.15.110:0/2632807052 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278009400 con 0x7f628c0b4680 2024-01-27T03:43:34.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.817+0000 7f628bfff640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4680 0x7f628c138d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6278000c00 tx=0x7f62780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.817+0000 7f62897fa640 10 client.?.objecter ms_handle_connect 0x7f628c0b4680 2024-01-27T03:43:34.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.817+0000 7f62897fa640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 1 -- 172.21.15.110:0/2632807052 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278032a40 con 0x7f628c0b4680 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 10 monclient: dump: 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.824 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 10 monclient: _finish_auth 0 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 10 monclient: _check_auth_tickets 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f6292d33640 5 monclient: authenticate success, global_id 4360 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 1 -- 172.21.15.110:0/2632807052 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6278032bc0 con 0x7f628c0b4680 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 1 -- 172.21.15.110:0/2632807052 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62780453f0 con 0x7f628c0b4680 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:34.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f62897fa640 10 monclient: dump: 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f6292d33640 10 monclient: _renew_subs 2024-01-27T03:43:34.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f628affd640 4 set_mon_vals no callback set 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f6292d33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c1396f0 con 0x7f628c0b4680 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f6292d33640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f6292d33640 10 monclient: _renew_subs 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f6292d33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.820+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c0aa070 con 0x7f628c0b4680 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.821+0000 7f6292d33640 1 librados: init done 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.822+0000 7f6292d33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.822+0000 7f6292d33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.822+0000 7f6292d33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:34.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.822+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c074d10 con 0x7f628c0b4680 2024-01-27T03:43:34.831 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-27T03:43:34.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.830+0000 7f38a3b73640 10 monclient: get_monmap_and_config 2024-01-27T03:43:34.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.830+0000 7f38a3b73640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.830+0000 7f38a3b73640 10 monclient: monmap: 2024-01-27T03:43:34.832 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.832 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:34.831950+0000 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:34.831950+0000 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.831+0000 7f38a3b73640 1 Processor -- start 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.831+0000 7f38a3b73640 1 -- start start 2024-01-27T03:43:34.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.831+0000 7f38a3b73640 10 monclient: init 2024-01-27T03:43:34.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-01-27T03:43:34.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f389c0b4b00 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:34.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f389c13f730 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:34.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f389c0b3db0 0x7f389c142770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 10 monclient(hunting): picked mon.noname-c con 0x7f389c0b3db0 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:34.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c0aaf30 con 0x7f389c0b4b00 2024-01-27T03:43:34.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c0ab210 con 0x7f389c13f730 2024-01-27T03:43:34.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389c0b44d0 con 0x7f389c0b3db0 2024-01-27T03:43:34.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.832+0000 7f38a3b73640 10 monclient(hunting): authenticate will time out at 1498.186374s 2024-01-27T03:43:34.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.841+0000 7f62897fa640 1 -- 172.21.15.110:0/2632807052 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f627804e450 con 0x7f628c0b4680 2024-01-27T03:43:34.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.841+0000 7f62897fa640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f627006ca20 0x7f627006eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.842+0000 7f6290acf640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f627006ca20 0x7f627006eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.842+0000 7f6290acf640 10 monclient: get_auth_request con 0x7f627006ca20 auth_method 0 2024-01-27T03:43:34.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.842+0000 7f62897fa640 10 client.4360.objecter ms_handle_connect 0x7f627006ca20 2024-01-27T03:43:34.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.842+0000 7f6290acf640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f627006ca20 0x7f627006eee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f62800015f0 tx=0x7f6280007a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:34.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.932+0000 7fcb81456640 10 monclient: get_monmap_and_config 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.932+0000 7fcb81456640 10 monclient: build_initial_monmap 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.932+0000 7fcb81456640 10 monclient: monmap: 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:34.933438+0000 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:34.933438+0000 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:34.934 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:34.935 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:34.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.932+0000 7fcb81456640 1 Processor -- start 2024-01-27T03:43:34.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 1 -- start start 2024-01-27T03:43:34.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient: init 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcb7c0b4680 0x7fcb7c142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient(hunting): picked mon.noname-a con 0x7fcb7c0b4680 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb7c0b4a60 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c142eb0 0x7fcb7c0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient(hunting): picked mon.noname-b con 0x7fcb7c142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c0afbe0 con 0x7fcb7c0b4a60 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c142240 con 0x7fcb7c0b4680 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:34.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0b0890 con 0x7fcb7c142eb0 2024-01-27T03:43:34.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:34.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.933+0000 7fcb81456640 10 monclient(hunting): authenticate will time out at 1498.287628s 2024-01-27T03:43:34.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.983+0000 7ff146ffd640 1 -- 172.21.15.110:0/2078080876 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7ff148054980 con 0x7ff1501487d0 2024-01-27T03:43:34.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.983+0000 7ff146ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.983+0000 7ff146ffd640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:34.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.994+0000 7f7b29ffb640 1 -- 172.21.15.110:0/1082160130 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f7b34057af0 con 0x7f7b380b3b30 2024-01-27T03:43:34.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.994+0000 7f7b29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:34.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.994+0000 7f7b29ffb640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:34.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.999+0000 7fb3cffff640 1 --2- 172.21.15.110:0/1759675993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4a60 0x7fb3d00b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.999+0000 7fcb7b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c142eb0 0x7fcb7c0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:34.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.999+0000 7fb3cffff640 10 monclient(hunting): get_auth_request con 0x7fb3d00b4a60 auth_method 0 2024-01-27T03:43:34.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.999+0000 7fb3cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:34.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.999+0000 7fb3cffff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:34.999+0000 7fb3cffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.000+0000 7fcb7b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c142eb0 0x7fcb7c0b1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38626/0 (socket says 172.21.15.110:38626) 2024-01-27T03:43:35.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.000+0000 7fcb7b7fe640 1 -- 172.21.15.110:0/1265963548 learned_addr learned my addr 172.21.15.110:0/1265963548 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:35.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.000+0000 7fcb7b7fe640 10 monclient(hunting): get_auth_request con 0x7fcb7c142eb0 auth_method 0 2024-01-27T03:43:35.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.000+0000 7fcb7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.000+0000 7fcb7b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.000+0000 7fcb7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.001+0000 7fb3cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.001+0000 7fb3cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.001+0000 7fb3cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fb3cffff640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-01-27T03:43:35.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fb3cffff640 1 -- 172.21.15.110:0/1759675993 >> v2:172.21.15.110:3300/0 conn(0x7fb3d00b3b30 msgr2=0x7fb3d00b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:35.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fb3cffff640 1 --2- 172.21.15.110:0/1759675993 >> v2:172.21.15.110:3300/0 conn(0x7fb3d00b3b30 0x7fb3d00b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fb3cffff640 1 -- 172.21.15.110:0/1759675993 >> v1:172.21.15.162:6789/0 conn(0x7fb3d00b4680 legacy=0x7fb3d0142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fb3cffff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:35.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fb3cffff640 1 monclient: found mon.noname-b 2024-01-27T03:43:35.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fb3cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:35.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fb3cffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:35.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fb3cffff640 1 -- 172.21.15.110:0/1759675993 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d00b0b00 con 0x7fb3d00b4a60 2024-01-27T03:43:35.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.003+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.006+0000 7fc0a8234640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-27T03:43:35.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.006+0000 7fc0a8234640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-27T03:43:35.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.006+0000 7fc0a8234640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.006+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fc0a00b6a70 con 0x7fc0a00b4be0 2024-01-27T03:43:35.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.007+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-01-27T03:43:35.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.007+0000 7fcb7b7fe640 1 -- 172.21.15.110:0/1265963548 >> v2:172.21.15.110:3300/0 conn(0x7fcb7c0b4680 msgr2=0x7fcb7c142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:35.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.007+0000 7fcb7b7fe640 1 --2- 172.21.15.110:0/1265963548 >> v2:172.21.15.110:3300/0 conn(0x7fcb7c0b4680 0x7fcb7c142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.007+0000 7fcb7b7fe640 1 -- 172.21.15.110:0/1265963548 >> v1:172.21.15.110:6789/0 conn(0x7fcb7c0b4a60 legacy=0x7fcb7c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:35.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.007+0000 7fcb7b7fe640 10 monclient: _finish_hunting 0 2024-01-27T03:43:35.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.007+0000 7fcb7b7fe640 1 monclient: found mon.noname-b 2024-01-27T03:43:35.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.007+0000 7fcb7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:35.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.007+0000 7fcb7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:35.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.007+0000 7fcb7b7fe640 1 -- 172.21.15.110:0/1265963548 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0b0b00 con 0x7fcb7c142eb0 2024-01-27T03:43:35.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.009+0000 7fb3cffff640 1 --2- 172.21.15.110:0/1759675993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4a60 0x7fb3d00b1470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0097a0 tx=0x7fb3bc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94ff9241d9ba7175 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.010+0000 7fcb7b7fe640 1 --2- 172.21.15.110:0/1265963548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c142eb0 0x7fcb7c0b1b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb74009e50 tx=0x7fcb7402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fd039173f69f6c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.015+0000 7fb3cf7fe640 1 -- 172.21.15.110:0/1759675993 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc031780 con 0x7fb3d00b4a60 2024-01-27T03:43:35.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.015+0000 7fb3cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.015+0000 7fb3cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.015+0000 7fb3cf7fe640 10 monclient: dump: 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.015+0000 7fb3cf7fe640 10 monclient: _finish_auth 0 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.015+0000 7fb3cf7fe640 10 monclient: _check_auth_tickets 2024-01-27T03:43:35.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.015+0000 7fb3cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:35.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.015+0000 7fb3d6ba9640 5 monclient: authenticate success, global_id 4316 2024-01-27T03:43:35.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.015+0000 7fb3d6ba9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:35.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.017+0000 7fcb79ffb640 1 -- 172.21.15.110:0/1265963548 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7402c9e0 con 0x7fcb7c142eb0 2024-01-27T03:43:35.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.017+0000 7fcb79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.017+0000 7fcb79ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.017+0000 7fcb79ffb640 10 monclient: dump: 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.017+0000 7fcb79ffb640 10 monclient: _finish_auth 0 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.017+0000 7fcb79ffb640 10 monclient: _check_auth_tickets 2024-01-27T03:43:35.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.017+0000 7fcb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:35.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.017+0000 7fcb81456640 5 monclient: authenticate success, global_id 4319 2024-01-27T03:43:35.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.017+0000 7fcb81456640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:35.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.069+0000 7f62897fa640 1 -- 172.21.15.110:0/2632807052 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f627807b080 con 0x7f628c0b4680 2024-01-27T03:43:35.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.069+0000 7f62897fa640 10 client.4360.objecter ms_dispatch 0x7f628c000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:35.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.069+0000 7f62897fa640 3 client.4360.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:35.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.069+0000 7f62897fa640 3 client.4360.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:35.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.069+0000 7f62897fa640 20 client.4360.objecter dump_active .. 0 homeless 2024-01-27T03:43:35.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.076+0000 7fcb79ffb640 1 -- 172.21.15.110:0/1265963548 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7402cb40 con 0x7fcb7c142eb0 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.076+0000 7fcb79ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.076+0000 7fcb79ffb640 1 -- 172.21.15.110:0/1265963548 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb740343f0 con 0x7fcb7c142eb0 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.076+0000 7fcb79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.076+0000 7fcb79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.076+0000 7fcb79ffb640 10 monclient: dump: 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.077 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:35.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 4 set_mon_vals no callback set 2024-01-27T03:43:35.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fb3cf7fe640 1 -- 172.21.15.110:0/1759675993 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc0318e0 con 0x7fb3d00b4a60 2024-01-27T03:43:35.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fb3cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:35.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fb3cf7fe640 1 -- 172.21.15.110:0/1759675993 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc031c70 con 0x7fb3d00b4a60 2024-01-27T03:43:35.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fb3cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fb3cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fb3cf7fe640 10 monclient: dump: 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.079 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fb3d6ba9640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:35.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fb3d6ba9640 4 set_mon_vals no callback set 2024-01-27T03:43:35.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 10 monclient: shutdown 2024-01-27T03:43:35.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 1 -- 172.21.15.110:0/1265963548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c142eb0 msgr2=0x7fcb7c0b1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 1 --2- 172.21.15.110:0/1265963548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c142eb0 0x7fcb7c0b1b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb74009e50 tx=0x7fcb7402be40 comp rx=0 tx=0).stop 2024-01-27T03:43:35.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 1 -- 172.21.15.110:0/1265963548 shutdown_connections 2024-01-27T03:43:35.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 1 --2- 172.21.15.110:0/1265963548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c142eb0 0x7fcb7c0b1b80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 1 --2- 172.21.15.110:0/1265963548 >> v2:172.21.15.110:3300/0 conn(0x7fcb7c0b4680 0x7fcb7c142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 1 -- 172.21.15.110:0/1265963548 >> 172.21.15.110:0/1265963548 conn(0x7fcb7c066490 msgr2=0x7fcb7c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 1 -- 172.21.15.110:0/1265963548 shutdown_connections 2024-01-27T03:43:35.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.077+0000 7fcb81456640 1 -- 172.21.15.110:0/1265963548 wait complete. 2024-01-27T03:43:35.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 10 monclient: shutdown 2024-01-27T03:43:35.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/1759675993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4a60 msgr2=0x7fb3d00b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 1 --2- 172.21.15.110:0/1759675993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4a60 0x7fb3d00b1470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0097a0 tx=0x7fb3bc02b820 comp rx=0 tx=0).stop 2024-01-27T03:43:35.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/1759675993 shutdown_connections 2024-01-27T03:43:35.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 1 --2- 172.21.15.110:0/1759675993 >> v2:172.21.15.110:3300/0 conn(0x7fb3d00b3b30 0x7fb3d00b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 1 --2- 172.21.15.110:0/1759675993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4a60 0x7fb3d00b1470 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/1759675993 >> 172.21.15.110:0/1759675993 conn(0x7fb3d0066490 msgr2=0x7fb3d00668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/1759675993 shutdown_connections 2024-01-27T03:43:35.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/1759675993 wait complete. 2024-01-27T03:43:35.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fcb81456640 10 monclient: build_initial_monmap 2024-01-27T03:43:35.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7f62897fa640 1 -- 172.21.15.110:0/2632807052 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f6278076c20 con 0x7f628c0b4680 2024-01-27T03:43:35.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7f62897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7f62897fa640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.078+0000 7fcb81456640 10 monclient: monmap: 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:35.084 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fcb81456640 1 librados: starting msgr at 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fcb81456640 1 librados: starting objecter 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fb3d6ba9640 10 monclient: build_initial_monmap 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fb3d6ba9640 10 monclient: monmap: 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:35.085 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:35.086 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:35.086 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fcb81456640 1 Processor -- start 2024-01-27T03:43:35.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fb3d6ba9640 1 librados: starting msgr at 2024-01-27T03:43:35.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fb3d6ba9640 1 librados: starting objecter 2024-01-27T03:43:35.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fcb81456640 1 -- start start 2024-01-27T03:43:35.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fcb81456640 1 librados: setting wanted keys 2024-01-27T03:43:35.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fcb81456640 1 librados: calling monclient init 2024-01-27T03:43:35.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.079+0000 7fcb81456640 10 monclient: init 2024-01-27T03:43:35.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 Processor -- start 2024-01-27T03:43:35.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 -- start start 2024-01-27T03:43:35.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 librados: setting wanted keys 2024-01-27T03:43:35.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 librados: calling monclient init 2024-01-27T03:43:35.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient: init 2024-01-27T03:43:35.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:35.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T03:43:35.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb7c0b4680 0x7fcb7c0b8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient(hunting): picked mon.noname-a con 0x7fcb7c0b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:35.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb7c142eb0 0x7fcb7c1deb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient(hunting): picked mon.noname-b con 0x7fcb7c142eb0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:35.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c1df080 0x7fcb7c13c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient(hunting): picked mon.noname-c con 0x7fcb7c1df080 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:35.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c13ecc0 con 0x7fcb7c0b4680 2024-01-27T03:43:35.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c13e970 con 0x7fcb7c142eb0 2024-01-27T03:43:35.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c06deb0 con 0x7fcb7c1df080 2024-01-27T03:43:35.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:35.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fcb81456640 10 monclient(hunting): authenticate will time out at 1498.433697s 2024-01-27T03:43:35.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3d00b3b30 0x7fb3d013cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient(hunting): picked mon.noname-a con 0x7fb3d00b3b30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4680 0x7fb3d013d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient(hunting): picked mon.noname-c con 0x7fb3d00b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d0139df0 0x7fb3d013a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient(hunting): picked mon.noname-b con 0x7fb3d0139df0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d013ec10 con 0x7fb3d00b3b30 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d013da70 con 0x7fb3d0139df0 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d013edf0 con 0x7fb3d00b4680 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:35.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.080+0000 7fb3d6ba9640 10 monclient(hunting): authenticate will time out at 1498.433892s 2024-01-27T03:43:35.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.083+0000 7f38a2110640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f389c0b3db0 0x7f389c142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.084+0000 7fb3d5146640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d0139df0 0x7fb3d013a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.084+0000 7fcb7b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c1df080 0x7fcb7c13c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.084+0000 7f38a2110640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f389c0b3db0 0x7f389c142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40846/0 (socket says 172.21.15.110:40846) 2024-01-27T03:43:35.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.084+0000 7f38a2110640 1 -- 172.21.15.110:0/2681399913 learned_addr learned my addr 172.21.15.110:0/2681399913 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:35.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.084+0000 7f38a2110640 10 monclient(hunting): get_auth_request con 0x7f389c0b3db0 auth_method 0 2024-01-27T03:43:35.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.084+0000 7f38a2110640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.084+0000 7f38a2110640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.084+0000 7f38a2110640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7fcb7b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c1df080 0x7fcb7c13c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38638/0 (socket says 172.21.15.110:38638) 2024-01-27T03:43:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7fcb7b7fe640 1 -- 172.21.15.110:0/2872394573 learned_addr learned my addr 172.21.15.110:0/2872394573 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7fcb7b7fe640 10 monclient(hunting): get_auth_request con 0x7fcb7c1df080 auth_method 0 2024-01-27T03:43:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7fcb7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7fcb7b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7fcb7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7f38a2110640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7f38a2110640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7f38a2110640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.085+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.086+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.086+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.086+0000 7fb3cffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4680 0x7fb3d013d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.086+0000 7fb3d5146640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d0139df0 0x7fb3d013a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40860/0 (socket says 172.21.15.110:40860) 2024-01-27T03:43:35.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.086+0000 7fb3d5146640 1 -- 172.21.15.110:0/2444737033 learned_addr learned my addr 172.21.15.110:0/2444737033 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:35.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.086+0000 7fb3d5146640 10 monclient(hunting): get_auth_request con 0x7fb3d0139df0 auth_method 0 2024-01-27T03:43:35.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.086+0000 7fb3d5146640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.086+0000 7fb3d5146640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.086+0000 7fb3d5146640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.087+0000 7fb3d5146640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.087+0000 7fb3d5146640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.087+0000 7fb3d5146640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-01-27T03:43:35.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 1 -- 172.21.15.110:0/2872394573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb7c142eb0 msgr2=0x7fcb7c1deb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb7c142eb0 0x7fcb7c1deb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 1 -- 172.21.15.110:0/2872394573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb7c0b4680 msgr2=0x7fcb7c0b8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb7c0b4680 0x7fcb7c0b8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 10 monclient: _finish_hunting 0 2024-01-27T03:43:35.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 1 monclient: found mon.noname-c 2024-01-27T03:43:35.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:35.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:43:35.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fcb7b7fe640 1 -- 172.21.15.110:0/2872394573 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb74009b30 con 0x7fcb7c1df080 2024-01-27T03:43:35.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fb3cffff640 10 monclient(hunting): get_auth_request con 0x7fb3d00b4680 auth_method 0 2024-01-27T03:43:35.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fb3cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fb3cffff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.089+0000 7fb3cffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.093+0000 7fb3cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.093+0000 7fb3cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.093+0000 7fb3cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.093+0000 7fcb7b7fe640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c1df080 0x7fcb7c13c0f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcb74009f70 tx=0x7fcb74004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.093+0000 7fcb637fe640 10 client.?.objecter ms_handle_connect 0x7fcb7c1df080 2024-01-27T03:43:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.093+0000 7fcb637fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2024-01-27T03:43:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 1 -- 172.21.15.110:0/2444737033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4680 msgr2=0x7fb3d013d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4680 0x7fb3d013d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 1 -- 172.21.15.110:0/2444737033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3d00b3b30 msgr2=0x7fb3d013cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3d00b3b30 0x7fb3d013cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 10 monclient: _finish_hunting 0 2024-01-27T03:43:35.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 1 monclient: found mon.noname-b 2024-01-27T03:43:35.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:35.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.094+0000 7fb3d5146640 1 -- 172.21.15.110:0/2444737033 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc009400 con 0x7fb3d0139df0 2024-01-27T03:43:35.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.095+0000 7f38a2110640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2024-01-27T03:43:35.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.095+0000 7f38a2110640 1 -- 172.21.15.110:0/2681399913 >> v1:172.21.15.162:6789/0 conn(0x7f389c13f730 legacy=0x7f389c142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:35.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.095+0000 7f38a2110640 1 -- 172.21.15.110:0/2681399913 >> v1:172.21.15.110:6789/0 conn(0x7f389c0b4b00 legacy=0x7f389c13f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.096+0000 7f38a2110640 10 monclient: _finish_hunting 0 2024-01-27T03:43:35.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.096+0000 7f38a2110640 1 monclient: found mon.noname-c 2024-01-27T03:43:35.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.096+0000 7f38a2110640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:35.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.096+0000 7f38a2110640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.096+0000 7f38a2110640 1 -- 172.21.15.110:0/2681399913 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c142cb0 con 0x7f389c0b3db0 2024-01-27T03:43:35.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.101+0000 7fb3d5146640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d0139df0 0x7fb3d013a270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800b9a0 tx=0x7fb3c800be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.101+0000 7fb3cd7fa640 10 client.?.objecter ms_handle_connect 0x7fb3d0139df0 2024-01-27T03:43:35.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.101+0000 7fb3cd7fa640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb637fe640 1 -- 172.21.15.110:0/2872394573 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7403e450 con 0x7fcb7c1df080 2024-01-27T03:43:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb637fe640 10 monclient: dump: 2024-01-27T03:43:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb637fe640 10 monclient: _finish_auth 0 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb637fe640 10 monclient: _check_auth_tickets 2024-01-27T03:43:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb637fe640 1 -- 172.21.15.110:0/2872394573 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7403e5d0 con 0x7fcb7c1df080 2024-01-27T03:43:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb81456640 5 monclient: authenticate success, global_id 4322 2024-01-27T03:43:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb81456640 10 monclient: _renew_subs 2024-01-27T03:43:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb81456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c06e190 con 0x7fcb7c1df080 2024-01-27T03:43:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb637fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb81456640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7fcb797fa640 4 set_mon_vals no callback set 2024-01-27T03:43:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.105+0000 7f38a2110640 1 --2- 172.21.15.110:0/2681399913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f389c0b3db0 0x7f389c142770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3894009e50 tx=0x7f389402be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f2568c1656d1719 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.106+0000 7fcb81456640 10 monclient: _renew_subs 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.106+0000 7fcb81456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.106+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c13f0e0 con 0x7fcb7c1df080 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.106+0000 7fcb81456640 1 librados: init done 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.107+0000 7fcb81456640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.107+0000 7fcb81456640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.107+0000 7fcb81456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.107+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c074d10 con 0x7fcb7c1df080 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.107+0000 7fcb637fe640 1 -- 172.21.15.110:0/2872394573 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7403e980 con 0x7fcb7c1df080 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.107+0000 7fcb637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.107+0000 7fcb637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.107+0000 7fcb637fe640 10 monclient: dump: 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.109 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.160+0000 7fc0967fc640 1 -- 172.21.15.110:0/413481326 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc09c04e090 con 0x7fc0a00b4be0 2024-01-27T03:43:35.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.160+0000 7fc0967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-27T03:43:35.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.160+0000 7fc0967fc640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:35.161 INFO:teuthology.orchestra.run.smithi110.stdout:55834574854 2024-01-27T03:43:35.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.162+0000 7fcb637fe640 1 -- 172.21.15.110:0/2872394573 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fcb7403eae0 con 0x7fcb7c1df080 2024-01-27T03:43:35.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fcb637fe640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fcb6806c9f0 0x7fcb6806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 10 librados: watch_flush enter 2024-01-27T03:43:35.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 10 librados: watch_flush exit 2024-01-27T03:43:35.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 20 client.4354.objecter shutdown clearing up homeless session... 2024-01-27T03:43:35.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 10 client.4354.objecter successfully canceled tick 2024-01-27T03:43:35.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fc08406c9a0 msgr2=0x7fc08406ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fc08406c9a0 0x7fc08406ee60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0136b30 tx=0x7fc090009040 comp rx=0 tx=0).stop 2024-01-27T03:43:35.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 10 monclient: shutdown 2024-01-27T03:43:35.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0a00b4be0 msgr2=0x7fc0a0142070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.163+0000 7fc0a8234640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0a00b4be0 0x7fc0a0142070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc09c033000 tx=0x7fc09c004a40 comp rx=0 tx=0).stop 2024-01-27T03:43:35.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fcb7affd640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fcb6806c9f0 0x7fcb6806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 shutdown_connections 2024-01-27T03:43:35.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fc0a8234640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fc08406c9a0 0x7fc08406ee60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fcb7affd640 10 monclient: get_auth_request con 0x7fcb6806c9f0 auth_method 0 2024-01-27T03:43:35.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fc0a8234640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0a013bc80 0x7fc0a00ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fc0a8234640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0a013c0e0 0x7fc0a0139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fc0a8234640 1 --2- 172.21.15.110:0/413481326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0a00b4be0 0x7fc0a0142070 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 >> 172.21.15.110:0/413481326 conn(0x7fc0a0066490 msgr2=0x7fc0a013b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 shutdown_connections 2024-01-27T03:43:35.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fc0a8234640 1 -- 172.21.15.110:0/413481326 wait complete. 2024-01-27T03:43:35.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fc0a8234640 1 librados: shutdown 2024-01-27T03:43:35.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fcb7affd640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fcb6806c9f0 0x7fcb6806eeb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009710 tx=0x7fcb6c006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.164+0000 7fcb637fe640 10 client.4322.objecter ms_handle_connect 0x7fcb6806c9f0 2024-01-27T03:43:35.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 1 -- 172.21.15.110:0/2681399913 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389402c9f0 con 0x7f389c0b3db0 2024-01-27T03:43:35.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:35.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 10 monclient: dump: 2024-01-27T03:43:35.169 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.169 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.169 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.169 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.169 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 10 monclient: _finish_auth 0 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 10 monclient: _check_auth_tickets 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 1 -- 172.21.15.110:0/2681399913 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389402cb50 con 0x7f389c0b3db0 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 1 -- 172.21.15.110:0/2681399913 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38940343f0 con 0x7f389c0b3db0 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:35.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a090d640 10 monclient: dump: 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a3b73640 5 monclient: authenticate success, global_id 4363 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a3b73640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:35.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.168+0000 7f38a3b73640 4 set_mon_vals no callback set 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 1 -- 172.21.15.110:0/2444737033 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c800cba0 con 0x7fb3d0139df0 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 10 monclient: dump: 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.173 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 10 monclient: _finish_auth 0 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 10 monclient: _check_auth_tickets 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 1 -- 172.21.15.110:0/2444737033 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c800cd20 con 0x7fb3d0139df0 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 1 -- 172.21.15.110:0/2444737033 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8014680 con 0x7fb3d0139df0 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3cd7fa640 10 monclient: dump: 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.174 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3ceffd640 4 set_mon_vals no callback set 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3d6ba9640 5 monclient: authenticate success, global_id 4366 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3d6ba9640 10 monclient: _renew_subs 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3d6ba9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d013f0d0 con 0x7fb3d0139df0 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3d6ba9640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:35.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3d6ba9640 10 monclient: _renew_subs 2024-01-27T03:43:35.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3d6ba9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d013df00 con 0x7fb3d0139df0 2024-01-27T03:43:35.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7fb3d6ba9640 1 librados: init done 2024-01-27T03:43:35.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7f38a3b73640 10 monclient: shutdown 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7f38a3b73640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7f38a3b73640 1 -- 172.21.15.110:0/2681399913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f389c0b3db0 msgr2=0x7f389c142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7f38a3b73640 1 --2- 172.21.15.110:0/2681399913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f389c0b3db0 0x7f389c142770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3894009e50 tx=0x7f389402be50 comp rx=0 tx=0).stop 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7f38a3b73640 1 -- 172.21.15.110:0/2681399913 shutdown_connections 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.169+0000 7f38a3b73640 1 --2- 172.21.15.110:0/2681399913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f389c0b3db0 0x7f389c142770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.170+0000 7f38a3b73640 1 -- 172.21.15.110:0/2681399913 >> 172.21.15.110:0/2681399913 conn(0x7f389c066490 msgr2=0x7f389c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.170+0000 7fb3d6ba9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.170+0000 7fb3d6ba9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.170+0000 7fb3d6ba9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.170+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d0074d10 con 0x7fb3d0139df0 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.170+0000 7f38a3b73640 1 -- 172.21.15.110:0/2681399913 shutdown_connections 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.170+0000 7f38a3b73640 1 -- 172.21.15.110:0/2681399913 wait complete. 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.171+0000 7f38a3b73640 10 monclient: build_initial_monmap 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.171+0000 7f38a3b73640 10 monclient: monmap: 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:35.181 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.172+0000 7f38a3b73640 1 librados: starting msgr at 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.172+0000 7f38a3b73640 1 librados: starting objecter 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.172+0000 7f38a3b73640 1 Processor -- start 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.172+0000 7f38a3b73640 1 -- start start 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.172+0000 7f38a3b73640 1 librados: setting wanted keys 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.172+0000 7f38a3b73640 1 librados: calling monclient init 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.172+0000 7f38a3b73640 10 monclient: init 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.173+0000 7f38a3b73640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.173+0000 7f38a3b73640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f389c0b3db0 0x7f389c0abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 10 monclient(hunting): picked mon.noname-c con 0x7f389c0b3db0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f389c0b4b00 0x7f389c0ac7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 10 monclient(hunting): picked mon.noname-b con 0x7f389c0b4b00 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f389c13f730 0x7f389c0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 10 monclient(hunting): picked mon.noname-a con 0x7f389c13f730 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c138e90 con 0x7f389c13f730 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c0adef0 con 0x7f389c0b4b00 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.174+0000 7f38a3b73640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a3b73640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389c0ae1d0 con 0x7f389c0b3db0 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a3b73640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a3b73640 10 monclient(hunting): authenticate will time out at 1498.528148s 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a2110640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f389c13f730 0x7f389c0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a2110640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f389c13f730 0x7f389c0acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50288/0 (socket says 172.21.15.110:50288) 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a2110640 1 -- 172.21.15.110:0/3570120980 learned_addr learned my addr 172.21.15.110:0/3570120980 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a2110640 10 monclient(hunting): get_auth_request con 0x7f389c13f730 auth_method 0 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a2110640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a2110640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.175+0000 7f38a2110640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.178+0000 7f38a2110640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.178+0000 7f38a2110640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.178+0000 7f38a2110640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-01-27T03:43:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-01-27T03:43:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 1 -- 172.21.15.110:0/3570120980 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f389c0b3db0 msgr2=0x7f389c0abde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f389c0b3db0 0x7f389c0abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 1 -- 172.21.15.110:0/3570120980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f389c0b4b00 msgr2=0x7f389c0ac7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f389c0b4b00 0x7f389c0ac7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 10 monclient: _finish_hunting 0 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 1 monclient: found mon.noname-a 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.182+0000 7f38a2110640 1 -- 172.21.15.110:0/3570120980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3894009b30 con 0x7f389c13f730 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.183+0000 7f38a2110640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f389c13f730 0x7f389c0acd00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3894036040 tx=0x7f3894004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.183+0000 7f38927fc640 10 client.?.objecter ms_handle_connect 0x7f389c13f730 2024-01-27T03:43:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.184+0000 7f38927fc640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:35.186 DEBUG:teuthology.parallel:result is None 2024-01-27T03:43:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 1 -- 172.21.15.110:0/3570120980 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389403e450 con 0x7f389c13f730 2024-01-27T03:43:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T03:43:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 10 monclient: dump: 2024-01-27T03:43:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 1 -- 172.21.15.110:0/3570120980 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389403e5d0 con 0x7f389c13f730 2024-01-27T03:43:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 1 -- 172.21.15.110:0/3570120980 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389403e9c0 con 0x7f389c13f730 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38927fc640 10 monclient: dump: 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.189 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f3893fff640 4 set_mon_vals no callback set 2024-01-27T03:43:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38a3b73640 5 monclient: authenticate success, global_id 4377 2024-01-27T03:43:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38a3b73640 10 monclient: _renew_subs 2024-01-27T03:43:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38a3b73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38a3b73640 1 -- 172.21.15.110:0/3570120980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c0ac320 con 0x7f389c13f730 2024-01-27T03:43:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38a3b73640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38a3b73640 10 monclient: _renew_subs 2024-01-27T03:43:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38a3b73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:35.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38a3b73640 1 -- 172.21.15.110:0/3570120980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c0ae7d0 con 0x7f389c13f730 2024-01-27T03:43:35.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.187+0000 7f38a3b73640 1 librados: init done 2024-01-27T03:43:35.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.189+0000 7f387ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.189+0000 7f387ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.189+0000 7f387ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:35.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.189+0000 7f387ffff640 1 -- 172.21.15.110:0/3570120980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3878005080 con 0x7f389c13f730 2024-01-27T03:43:35.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.196+0000 7fcb637fe640 1 -- 172.21.15.110:0/2872394573 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcb7407c1c0 con 0x7fcb7c1df080 2024-01-27T03:43:35.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.196+0000 7fcb637fe640 10 client.4322.objecter ms_dispatch 0x7fcb7c000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:35.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.196+0000 7fcb637fe640 3 client.4322.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:35.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.196+0000 7fcb637fe640 3 client.4322.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:35.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.197+0000 7fcb637fe640 20 client.4322.objecter dump_active .. 0 homeless 2024-01-27T03:43:35.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.234+0000 7ff158705640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-27T03:43:35.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.234+0000 7ff158705640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-27T03:43:35.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.234+0000 7ff158705640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:35.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.234+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff150140ce0 con 0x7ff1501487d0 2024-01-27T03:43:35.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.279+0000 7f7b3f55e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-27T03:43:35.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.279+0000 7f7b3f55e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-27T03:43:35.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.279+0000 7f7b3f55e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:35.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.279+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7b380af760 con 0x7f7b380b3b30 2024-01-27T03:43:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.283+0000 7f6292d33640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-27T03:43:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.283+0000 7f6292d33640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-27T03:43:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.283+0000 7f6292d33640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.283+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f628c1436a0 con 0x7f628c0b4680 2024-01-27T03:43:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.422+0000 7fb3cd7fa640 1 -- 172.21.15.110:0/2444737033 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fb3c801d3b0 con 0x7fb3d0139df0 2024-01-27T03:43:35.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.422+0000 7fb3cd7fa640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.422+0000 7fb3d4945640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.423+0000 7fb3d4945640 10 monclient: get_auth_request con 0x7fb3b406c9f0 auth_method 0 2024-01-27T03:43:35.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.423+0000 7fb3d4945640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4000f60 tx=0x7fb3c4007a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.423+0000 7fb3cd7fa640 10 client.4366.objecter ms_handle_connect 0x7fb3b406c9f0 2024-01-27T03:43:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.427+0000 7fb3cd7fa640 1 -- 172.21.15.110:0/2444737033 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb3c8058780 con 0x7fb3d0139df0 2024-01-27T03:43:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.427+0000 7fb3cd7fa640 10 client.4366.objecter ms_dispatch 0x7fb3d0000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.427+0000 7fb3cd7fa640 3 client.4366.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.427+0000 7fb3cd7fa640 3 client.4366.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:35.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.427+0000 7fb3cd7fa640 20 client.4366.objecter dump_active .. 0 homeless 2024-01-27T03:43:35.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.437+0000 7fb3cd7fa640 1 -- 172.21.15.110:0/2444737033 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fb3c8054a20 con 0x7fb3d0139df0 2024-01-27T03:43:35.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.437+0000 7fb3cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.437+0000 7fb3cd7fa640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:35.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.438+0000 7f62897fa640 1 -- 172.21.15.110:0/2632807052 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f628c1436a0 con 0x7f628c0b4680 2024-01-27T03:43:35.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.438+0000 7f62897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-27T03:43:35.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.438+0000 7f62897fa640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:35.438 INFO:teuthology.orchestra.run.smithi110.stdout:68719476743 2024-01-27T03:43:35.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.440+0000 7fcb637fe640 1 -- 172.21.15.110:0/2872394573 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fcb74077d60 con 0x7fcb7c1df080 2024-01-27T03:43:35.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.440+0000 7fcb637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.440+0000 7fcb637fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:35.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 10 librados: watch_flush enter 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 10 librados: watch_flush exit 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 20 client.4360.objecter shutdown clearing up homeless session... 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 10 client.4360.objecter successfully canceled tick 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f627006ca20 msgr2=0x7f627006eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f627006ca20 0x7f627006eee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f62800015f0 tx=0x7f6280007a40 comp rx=0 tx=0).stop 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 10 monclient: shutdown 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4680 msgr2=0x7f628c138d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4680 0x7f628c138d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6278000c00 tx=0x7f62780046c0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 shutdown_connections 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f627006ca20 0x7f627006eee0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f628c139240 0x7f628c0a9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f628c0b4680 0x7f628c138d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 --2- 172.21.15.110:0/2632807052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f628c0b3b30 0x7f628c14ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 >> 172.21.15.110:0/2632807052 conn(0x7f628c066490 msgr2=0x7f628c13f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 shutdown_connections 2024-01-27T03:43:35.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 -- 172.21.15.110:0/2632807052 wait complete. 2024-01-27T03:43:35.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.441+0000 7f6292d33640 1 librados: shutdown 2024-01-27T03:43:35.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.445+0000 7f38927fc640 1 -- 172.21.15.110:0/3570120980 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f389403eb20 con 0x7f389c13f730 2024-01-27T03:43:35.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.446+0000 7f38927fc640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f388006c9f0 0x7f388006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.446+0000 7f38a190f640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f388006c9f0 0x7f388006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.446+0000 7f38a190f640 10 monclient: get_auth_request con 0x7f388006c9f0 auth_method 0 2024-01-27T03:43:35.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.447+0000 7f7b29ffb640 1 -- 172.21.15.110:0/1082160130 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7b34017030 con 0x7f7b380b3b30 2024-01-27T03:43:35.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.447+0000 7f7b29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-27T03:43:35.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.447+0000 7f7b29ffb640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:35.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.447+0000 7f38927fc640 10 client.4377.objecter ms_handle_connect 0x7f388006c9f0 2024-01-27T03:43:35.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.447+0000 7f38a190f640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f388006c9f0 0x7f388006eeb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3898000f60 tx=0x7f38980079b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.447 INFO:teuthology.orchestra.run.smithi110.stdout:64424509447 2024-01-27T03:43:35.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 10 librados: watch_flush enter 2024-01-27T03:43:35.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 10 librados: watch_flush exit 2024-01-27T03:43:35.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 20 client.4313.objecter shutdown clearing up homeless session... 2024-01-27T03:43:35.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 10 client.4313.objecter successfully canceled tick 2024-01-27T03:43:35.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7b1c06c9d0 msgr2=0x7f7b1c06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7b1c06c9d0 0x7f7b1c06ee90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c000c00 tx=0x7f7b2c03d040 comp rx=0 tx=0).stop 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 10 monclient: shutdown 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b3b30 msgr2=0x7f7b381d2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b3b30 0x7f7b381d2c30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400c430 tx=0x7f7b3400c8f0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 shutdown_connections 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7b1c06c9d0 0x7f7b1c06ee90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b381389f0 0x7f7b38138ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b380b4680 0x7f7b380ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 --2- 172.21.15.110:0/1082160130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b380b3b30 0x7f7b381d2c30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.449+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 >> 172.21.15.110:0/1082160130 conn(0x7f7b38066490 msgr2=0x7f7b380b57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.450+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 shutdown_connections 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.450+0000 7f7b3f55e640 1 -- 172.21.15.110:0/1082160130 wait complete. 2024-01-27T03:43:35.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.450+0000 7f7b3f55e640 1 librados: shutdown 2024-01-27T03:43:35.459 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.0 2024-01-27T03:43:35.459 DEBUG:teuthology.parallel:result is None 2024-01-27T03:43:35.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.5 2024-01-27T03:43:35.462 DEBUG:teuthology.parallel:result is None 2024-01-27T03:43:35.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.463+0000 7f38927fc640 1 -- 172.21.15.110:0/3570120980 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3894033e50 con 0x7f389c13f730 2024-01-27T03:43:35.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.463+0000 7f38927fc640 10 client.4377.objecter ms_dispatch 0x7f389c000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:35.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.463+0000 7f38927fc640 3 client.4377.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:35.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.463+0000 7f38927fc640 3 client.4377.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:35.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.463+0000 7f38927fc640 20 client.4377.objecter dump_active .. 0 homeless 2024-01-27T03:43:35.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.463+0000 7f38927fc640 1 -- 172.21.15.110:0/3570120980 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f3894077b70 con 0x7f389c13f730 2024-01-27T03:43:35.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.463+0000 7f38927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.463+0000 7f38927fc640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:35.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.472+0000 7ff146ffd640 1 -- 172.21.15.110:0/2078080876 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff148019070 con 0x7ff1501487d0 2024-01-27T03:43:35.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.472+0000 7ff146ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-27T03:43:35.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.472+0000 7ff146ffd640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:35.473 INFO:teuthology.orchestra.run.smithi110.stdout:85899345929 2024-01-27T03:43:35.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.474+0000 7ff158705640 10 librados: watch_flush enter 2024-01-27T03:43:35.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.474+0000 7ff158705640 10 librados: watch_flush exit 2024-01-27T03:43:35.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.474+0000 7ff158705640 20 client.4371.objecter shutdown clearing up homeless session... 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.474+0000 7ff158705640 10 client.4371.objecter successfully canceled tick 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff13406c9a0 msgr2=0x7ff13406ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff13406c9a0 0x7ff13406ee60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff14c000f60 tx=0x7ff14c0078f0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 10 monclient: shutdown 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501487d0 msgr2=0x7ff150135140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501487d0 0x7ff150135140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff14800b9e0 tx=0x7ff14800bea0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 shutdown_connections 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff13406c9a0 0x7ff13406ee60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1501403b0 0x7ff150134c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1501487d0 0x7ff150135140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 --2- 172.21.15.110:0/2078080876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff15006fc00 0x7ff15006d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 >> 172.21.15.110:0/2078080876 conn(0x7ff150066f00 msgr2=0x7ff150138fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 shutdown_connections 2024-01-27T03:43:35.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 -- 172.21.15.110:0/2078080876 wait complete. 2024-01-27T03:43:35.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.475+0000 7ff158705640 1 librados: shutdown 2024-01-27T03:43:35.489 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345929 for osd.4 2024-01-27T03:43:35.489 DEBUG:teuthology.parallel:result is None 2024-01-27T03:43:35.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.630+0000 7fb3d6ba9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-27T03:43:35.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.630+0000 7fb3d6ba9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-27T03:43:35.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.630+0000 7fb3d6ba9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.630+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb3d013e090 con 0x7fb3d0139df0 2024-01-27T03:43:35.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.634+0000 7fb3cd7fa640 1 -- 172.21.15.110:0/2444737033 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb3c8019020 con 0x7fb3d0139df0 2024-01-27T03:43:35.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.634+0000 7fb3cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-27T03:43:35.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.634+0000 7fb3cd7fa640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:35.634 INFO:teuthology.orchestra.run.smithi110.stdout:85899345931 2024-01-27T03:43:35.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 10 librados: watch_flush enter 2024-01-27T03:43:35.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 10 librados: watch_flush exit 2024-01-27T03:43:35.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 20 client.4366.objecter shutdown clearing up homeless session... 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 10 client.4366.objecter successfully canceled tick 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 msgr2=0x7fb3b406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4000f60 tx=0x7fb3c4007a80 comp rx=0 tx=0).stop 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 10 monclient: shutdown 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d0139df0 msgr2=0x7fb3d013a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d0139df0 0x7fb3d013a270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800b9a0 tx=0x7fb3c800be60 comp rx=0 tx=0).stop 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 shutdown_connections 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3d00b4680 0x7fb3d013d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3d0139df0 0x7fb3d013a270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 --2- 172.21.15.110:0/2444737033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3d00b3b30 0x7fb3d013cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 >> 172.21.15.110:0/2444737033 conn(0x7fb3d0066490 msgr2=0x7fb3d00b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 shutdown_connections 2024-01-27T03:43:35.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 -- 172.21.15.110:0/2444737033 wait complete. 2024-01-27T03:43:35.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.637+0000 7fb3d6ba9640 1 librados: shutdown 2024-01-27T03:43:35.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.647+0000 7fcb81456640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-27T03:43:35.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.647+0000 7fcb81456640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-27T03:43:35.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.647+0000 7fcb81456640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T03:43:35.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.647+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fcb7c13f270 con 0x7fcb7c1df080 2024-01-27T03:43:35.648 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345931 got 85899345931 for osd.6 2024-01-27T03:43:35.649 DEBUG:teuthology.parallel:result is None 2024-01-27T03:43:35.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.653+0000 7fcb637fe640 1 -- 172.21.15.110:0/2872394573 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcb74043090 con 0x7fcb7c1df080 2024-01-27T03:43:35.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.653+0000 7fcb637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-27T03:43:35.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.653+0000 7fcb637fe640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:35.655 INFO:teuthology.orchestra.run.smithi110.stdout:90194313225 2024-01-27T03:43:35.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 10 librados: watch_flush enter 2024-01-27T03:43:35.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 10 librados: watch_flush exit 2024-01-27T03:43:35.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 20 client.4322.objecter shutdown clearing up homeless session... 2024-01-27T03:43:35.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 10 client.4322.objecter successfully canceled tick 2024-01-27T03:43:35.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fcb6806c9f0 msgr2=0x7fcb6806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fcb6806c9f0 0x7fcb6806eeb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009710 tx=0x7fcb6c006c90 comp rx=0 tx=0).stop 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 10 monclient: shutdown 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c1df080 msgr2=0x7fcb7c13c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c1df080 0x7fcb7c13c0f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcb74009f70 tx=0x7fcb74004a40 comp rx=0 tx=0).stop 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 shutdown_connections 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fcb6806c9f0 0x7fcb6806eeb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb7c1df080 0x7fcb7c13c0f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb7c142eb0 0x7fcb7c1deb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 --2- 172.21.15.110:0/2872394573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb7c0b4680 0x7fcb7c0b8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 >> 172.21.15.110:0/2872394573 conn(0x7fcb7c066490 msgr2=0x7fcb7c13b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 shutdown_connections 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 -- 172.21.15.110:0/2872394573 wait complete. 2024-01-27T03:43:35.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.655+0000 7fcb81456640 1 librados: shutdown 2024-01-27T03:43:35.666 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313225 for osd.1 2024-01-27T03:43:35.667 DEBUG:teuthology.parallel:result is None 2024-01-27T03:43:35.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.669+0000 7f387ffff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-27T03:43:35.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.669+0000 7f387ffff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-27T03:43:35.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.669+0000 7f387ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T03:43:35.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.669+0000 7f387ffff640 1 -- 172.21.15.110:0/3570120980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3878005600 con 0x7f389c13f730 2024-01-27T03:43:35.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.672+0000 7f38927fc640 1 -- 172.21.15.110:0/3570120980 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3894043090 con 0x7f389c13f730 2024-01-27T03:43:35.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.672+0000 7f38927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-27T03:43:35.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.672+0000 7f38927fc640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:35.673 INFO:teuthology.orchestra.run.smithi110.stdout:68719476743 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 10 librados: watch_flush enter 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 10 librados: watch_flush exit 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 20 client.4377.objecter shutdown clearing up homeless session... 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 10 client.4377.objecter successfully canceled tick 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 -- 172.21.15.110:0/3570120980 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f388006c9f0 msgr2=0x7f388006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f388006c9f0 0x7f388006eeb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3898000f60 tx=0x7f38980079b0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 10 monclient: shutdown 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 -- 172.21.15.110:0/3570120980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f389c13f730 msgr2=0x7f389c0acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f389c13f730 0x7f389c0acd00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3894036040 tx=0x7f3894004a40 comp rx=0 tx=0).stop 2024-01-27T03:43:35.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 -- 172.21.15.110:0/3570120980 shutdown_connections 2024-01-27T03:43:35.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f388006c9f0 0x7f388006eeb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f389c13f730 0x7f389c0acd00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f389c0b4b00 0x7f389c0ac7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 --2- 172.21.15.110:0/3570120980 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f389c0b3db0 0x7f389c0abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 -- 172.21.15.110:0/3570120980 >> 172.21.15.110:0/3570120980 conn(0x7f389c066490 msgr2=0x7f389c0b72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 -- 172.21.15.110:0/3570120980 shutdown_connections 2024-01-27T03:43:35.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 -- 172.21.15.110:0/3570120980 wait complete. 2024-01-27T03:43:35.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.675+0000 7f387ffff640 1 librados: shutdown 2024-01-27T03:43:35.686 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.3 2024-01-27T03:43:35.686 DEBUG:teuthology.parallel:result is None 2024-01-27T03:43:35.686 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-27T03:43:35.686 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T03:43:35.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.824+0000 7fa24a790640 10 monclient: get_monmap_and_config 2024-01-27T03:43:35.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.824+0000 7fa24a790640 10 monclient: build_initial_monmap 2024-01-27T03:43:35.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.824+0000 7fa24a790640 10 monclient: monmap: 2024-01-27T03:43:35.824 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:35.824 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.824 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:35.824497+0000 2024-01-27T03:43:35.824 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:35.824497+0000 2024-01-27T03:43:35.824 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:35.824 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.825 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:35.825 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:35.825 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:35.825 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:35.825 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:35.825 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 1 Processor -- start 2024-01-27T03:43:35.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 1 -- start start 2024-01-27T03:43:35.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient: init 2024-01-27T03:43:35.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:35.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-01-27T03:43:35.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa24406bcb0 0x7fa24413f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient(hunting): picked mon.noname-c con 0x7fa24406bcb0 addr v2:172.21.15.162:3300/0 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa24413f910 0x7fa244145830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient(hunting): picked mon.noname-a con 0x7fa24413f910 addr v2:172.21.15.110:3300/0 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa244145d70 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24406cea0 con 0x7fa244145d70 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24406d180 con 0x7fa24413f910 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2440ae090 con 0x7fa24406bcb0 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.825+0000 7fa24a790640 10 monclient(hunting): authenticate will time out at 1499.178700s 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.826+0000 7fa243fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa24406bcb0 0x7fa24413f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.826+0000 7fa243fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa24406bcb0 0x7fa24413f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40874/0 (socket says 172.21.15.110:40874) 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.826+0000 7fa243fff640 1 -- 172.21.15.110:0/1095712141 learned_addr learned my addr 172.21.15.110:0/1095712141 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.826+0000 7fa243fff640 10 monclient(hunting): get_auth_request con 0x7fa24406bcb0 auth_method 0 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.826+0000 7fa243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.826+0000 7fa243fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.826+0000 7fa243fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.826+0000 7fa2437fe640 1 --2- 172.21.15.110:0/1095712141 >> v2:172.21.15.110:3300/0 conn(0x7fa24413f910 0x7fa244145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.827+0000 7fa243fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.827+0000 7fa243fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.827+0000 7fa243fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.827+0000 7fa2437fe640 10 monclient(hunting): get_auth_request con 0x7fa24413f910 auth_method 0 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.827+0000 7fa2437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.827+0000 7fa2437fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.827+0000 7fa2437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.828+0000 7fa243fff640 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2024-01-27T03:43:35.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.828+0000 7fa243fff640 1 -- 172.21.15.110:0/1095712141 >> v2:172.21.15.110:3300/0 conn(0x7fa24413f910 msgr2=0x7fa244145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:43:35.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.828+0000 7fa243fff640 1 --2- 172.21.15.110:0/1095712141 >> v2:172.21.15.110:3300/0 conn(0x7fa24413f910 0x7fa244145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.828+0000 7fa243fff640 1 -- 172.21.15.110:0/1095712141 >> v1:172.21.15.110:6789/0 conn(0x7fa244145d70 legacy=0x7fa244141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.828+0000 7fa243fff640 10 monclient: _finish_hunting 0 2024-01-27T03:43:35.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.828+0000 7fa243fff640 1 monclient: found mon.noname-c 2024-01-27T03:43:35.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.828+0000 7fa243fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:35.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.828+0000 7fa243fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.828+0000 7fa243fff640 1 -- 172.21.15.110:0/1095712141 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24406ca50 con 0x7fa24406bcb0 2024-01-27T03:43:35.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.830+0000 7fa243fff640 1 --2- 172.21.15.110:0/1095712141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa24406bcb0 0x7fa24413f3d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa2440b35f0 tx=0x7fa23402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7abdf016acc4f1cc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 1 -- 172.21.15.110:0/1095712141 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa234030860 con 0x7fa24406bcb0 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 10 monclient: dump: 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 10 monclient: _finish_auth 0 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 10 monclient: _check_auth_tickets 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 1 -- 172.21.15.110:0/1095712141 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2340309c0 con 0x7fa24406bcb0 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa24a790640 5 monclient: authenticate success, global_id 4369 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa24a790640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 1 -- 172.21.15.110:0/1095712141 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa234030d50 con 0x7fa24406bcb0 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa24a790640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa24a790640 4 set_mon_vals no callback set 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa242ffd640 10 monclient: dump: 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.836 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa24a790640 10 monclient: shutdown 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa24a790640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa24a790640 1 -- 172.21.15.110:0/1095712141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa24406bcb0 msgr2=0x7fa24413f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.834+0000 7fa24a790640 1 --2- 172.21.15.110:0/1095712141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa24406bcb0 0x7fa24413f3d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa2440b35f0 tx=0x7fa23402b9a0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 1 -- 172.21.15.110:0/1095712141 shutdown_connections 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 1 --2- 172.21.15.110:0/1095712141 >> v2:172.21.15.110:3300/0 conn(0x7fa24413f910 0x7fa244145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 1 --2- 172.21.15.110:0/1095712141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa24406bcb0 0x7fa24413f3d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 1 -- 172.21.15.110:0/1095712141 >> 172.21.15.110:0/1095712141 conn(0x7fa2440b2640 msgr2=0x7fa2440b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:35.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 1 -- 172.21.15.110:0/1095712141 shutdown_connections 2024-01-27T03:43:35.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 1 -- 172.21.15.110:0/1095712141 wait complete. 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 10 monclient: build_initial_monmap 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 10 monclient: monmap: 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 1 librados: starting msgr at 2024-01-27T03:43:35.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.835+0000 7fa24a790640 1 librados: starting objecter 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 Processor -- start 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 -- start start 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 librados: setting wanted keys 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 librados: calling monclient init 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient: init 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa24406bcb0 0x7fa2441dab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient(hunting): picked mon.noname-a con 0x7fa24406bcb0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa24413f910 0x7fa2441db430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient(hunting): picked mon.noname-c con 0x7fa24413f910 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa244145d70 0x7fa244070b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient(hunting): picked mon.noname-b con 0x7fa244145d70 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24413bbf0 con 0x7fa24406bcb0 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24413c7f0 con 0x7fa244145d70 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24413ca60 con 0x7fa24413f910 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.836+0000 7fa24a790640 10 monclient(hunting): authenticate will time out at 1499.189746s 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa243fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa24406bcb0 0x7fa2441dab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa248d2d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa244145d70 0x7fa244070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa2437fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa24413f910 0x7fa2441db430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa243fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa24406bcb0 0x7fa2441dab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50300/0 (socket says 172.21.15.110:50300) 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa243fff640 1 -- 172.21.15.110:0/46506201 learned_addr learned my addr 172.21.15.110:0/46506201 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa243fff640 10 monclient(hunting): get_auth_request con 0x7fa24406bcb0 auth_method 0 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa243fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa243fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa248d2d640 10 monclient(hunting): get_auth_request con 0x7fa244145d70 auth_method 0 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa248d2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa248d2d640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa248d2d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa2437fe640 10 monclient(hunting): get_auth_request con 0x7fa24413f910 auth_method 0 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa2437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa2437fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.837+0000 7fa2437fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.838+0000 7fa248d2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.838+0000 7fa248d2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.838+0000 7fa248d2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.838+0000 7fa243fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.838+0000 7fa243fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.838+0000 7fa243fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.838+0000 7fa2437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.838+0000 7fa2437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.838+0000 7fa2437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 1 -- 172.21.15.110:0/46506201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa24413f910 msgr2=0x7fa2441db430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa24413f910 0x7fa2441db430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 1 -- 172.21.15.110:0/46506201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa24406bcb0 msgr2=0x7fa2441dab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa24406bcb0 0x7fa2441dab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 10 monclient: _finish_hunting 0 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 1 monclient: found mon.noname-b 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.840+0000 7fa248d2d640 1 -- 172.21.15.110:0/46506201 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2340094d0 con 0x7fa244145d70 2024-01-27T03:43:35.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.842+0000 7fa248d2d640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa244145d70 0x7fa244070b40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00c700 tx=0x7fa23c00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.843+0000 7fa240ff9640 10 client.?.objecter ms_handle_connect 0x7fa244145d70 2024-01-27T03:43:35.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.843+0000 7fa240ff9640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:35.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 1 -- 172.21.15.110:0/46506201 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c003b70 con 0x7fa244145d70 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 10 monclient: dump: 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 10 monclient: _finish_auth 0 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 10 monclient: _check_auth_tickets 2024-01-27T03:43:35.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 1 -- 172.21.15.110:0/46506201 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c003cd0 con 0x7fa244145d70 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa24a790640 5 monclient: authenticate success, global_id 4372 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 1 -- 172.21.15.110:0/46506201 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c005310 con 0x7fa244145d70 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa24a790640 10 monclient: _renew_subs 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa24a790640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa244071080 con 0x7fa244145d70 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa2427fc640 4 set_mon_vals no callback set 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa240ff9640 10 monclient: dump: 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.848 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa24a790640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa24a790640 10 monclient: _renew_subs 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa24a790640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2441450d0 con 0x7fa244145d70 2024-01-27T03:43:35.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.846+0000 7fa24a790640 1 librados: init done 2024-01-27T03:43:35.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.847+0000 7fa24a790640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.847+0000 7fa24a790640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:35.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.847+0000 7fa24a790640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:35.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.847+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24406d430 con 0x7fa244145d70 2024-01-27T03:43:35.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.865+0000 7fa240ff9640 1 -- 172.21.15.110:0/46506201 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fa23c005470 con 0x7fa244145d70 2024-01-27T03:43:35.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.866+0000 7fa240ff9640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa22406ca20 0x7fa22406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:35.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.866+0000 7fa243fff640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa22406ca20 0x7fa22406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:35.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.866+0000 7fa243fff640 10 monclient: get_auth_request con 0x7fa22406ca20 auth_method 0 2024-01-27T03:43:35.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.867+0000 7fa243fff640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa22406ca20 0x7fa22406eee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa2340090b0 tx=0x7fa234008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:35.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:35.867+0000 7fa240ff9640 10 client.4372.objecter ms_handle_connect 0x7fa22406ca20 2024-01-27T03:43:36.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.094+0000 7fa240ff9640 1 -- 172.21.15.110:0/46506201 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa23c058960 con 0x7fa244145d70 2024-01-27T03:43:36.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.094+0000 7fa240ff9640 10 client.4372.objecter ms_dispatch 0x7fa244000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:36.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.094+0000 7fa240ff9640 3 client.4372.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:36.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.094+0000 7fa240ff9640 3 client.4372.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:36.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.094+0000 7fa240ff9640 20 client.4372.objecter dump_active .. 0 homeless 2024-01-27T03:43:36.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.104+0000 7fa240ff9640 1 -- 172.21.15.110:0/46506201 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fa23c054c00 con 0x7fa244145d70 2024-01-27T03:43:36.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.104+0000 7fa240ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:36.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.104+0000 7fa240ff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:36.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.289+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2440b0090 con 0x7fa22406ca20 2024-01-27T03:43:36.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.291+0000 7fa240ff9640 1 -- 172.21.15.110:0/46506201 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48063 (secure 0 0 0) 0x7fa234030bb0 con 0x7fa22406ca20 2024-01-27T03:43:36.292 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:43:36.292 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 10 librados: watch_flush enter 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 10 librados: watch_flush exit 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 20 client.4372.objecter shutdown clearing up homeless session... 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 10 client.4372.objecter successfully canceled tick 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa22406ca20 msgr2=0x7fa22406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa22406ca20 0x7fa22406eee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa2340090b0 tx=0x7fa234008030 comp rx=0 tx=0).stop 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 10 monclient: shutdown 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa244145d70 msgr2=0x7fa244070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa244145d70 0x7fa244070b40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00c700 tx=0x7fa23c00cbc0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 shutdown_connections 2024-01-27T03:43:36.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa22406ca20 0x7fa22406eee0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa24413f910 0x7fa2441db430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa244145d70 0x7fa244070b40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 --2- 172.21.15.110:0/46506201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa24406bcb0 0x7fa2441dab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.293+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 >> 172.21.15.110:0/46506201 conn(0x7fa2440b2640 msgr2=0x7fa2440ae6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:36.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.294+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 shutdown_connections 2024-01-27T03:43:36.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.294+0000 7fa24a790640 1 -- 172.21.15.110:0/46506201 wait complete. 2024-01-27T03:43:36.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.294+0000 7fa24a790640 1 librados: shutdown 2024-01-27T03:43:36.304 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-01-27T03:43:34.843402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218896,"kb_used_data":2712,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755824,"statfs":{"total":773094113280,"available":772869963776,"internally_reserved":0,"allocated":2777088,"data_stored":1482456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.337047"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.824345+0000","last_change":"2024-01-27T03:43:30.350497+0000","last_active":"2024-01-27T03:43:30.824345+0000","last_peered":"2024-01-27T03:43:30.824345+0000","last_clean":"2024-01-27T03:43:30.824345+0000","last_became_active":"2024-01-27T03:43:30.345375+0000","last_became_peered":"2024-01-27T03:43:30.345375+0000","last_unstale":"2024-01-27T03:43:30.824345+0000","last_undegraded":"2024-01-27T03:43:30.824345+0000","last_fullsized":"2024-01-27T03:43:30.824345+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:58.675410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.461956+0000","last_change":"2024-01-27T03:43:30.372081+0000","last_active":"2024-01-27T03:43:30.461956+0000","last_peered":"2024-01-27T03:43:30.461956+0000","last_clean":"2024-01-27T03:43:30.461956+0000","last_became_active":"2024-01-27T03:43:30.335129+0000","last_became_peered":"2024-01-27T03:43:30.335129+0000","last_unstale":"2024-01-27T03:43:30.461956+0000","last_undegraded":"2024-01-27T03:43:30.461956+0000","last_fullsized":"2024-01-27T03:43:30.461956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T14:26:22.021010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.818107+0000","last_change":"2024-01-27T03:43:30.077908+0000","last_active":"2024-01-27T03:43:30.818107+0000","last_peered":"2024-01-27T03:43:30.818107+0000","last_clean":"2024-01-27T03:43:30.818107+0000","last_became_active":"2024-01-27T03:43:30.057792+0000","last_became_peered":"2024-01-27T03:43:30.057792+0000","last_unstale":"2024-01-27T03:43:30.818107+0000","last_undegraded":"2024-01-27T03:43:30.818107+0000","last_fullsized":"2024-01-27T03:43:30.818107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:05.446156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.734027+0000","last_change":"2024-01-27T03:43:30.340472+0000","last_active":"2024-01-27T03:43:30.734027+0000","last_peered":"2024-01-27T03:43:30.734027+0000","last_clean":"2024-01-27T03:43:30.734027+0000","last_became_active":"2024-01-27T03:43:30.329560+0000","last_became_peered":"2024-01-27T03:43:30.329560+0000","last_unstale":"2024-01-27T03:43:30.734027+0000","last_undegraded":"2024-01-27T03:43:30.734027+0000","last_fullsized":"2024-01-27T03:43:30.734027+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:10:58.217001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.815054+0000","last_change":"2024-01-27T03:43:30.335032+0000","last_active":"2024-01-27T03:43:30.815054+0000","last_peered":"2024-01-27T03:43:30.815054+0000","last_clean":"2024-01-27T03:43:30.815054+0000","last_became_active":"2024-01-27T03:43:30.329967+0000","last_became_peered":"2024-01-27T03:43:30.329967+0000","last_unstale":"2024-01-27T03:43:30.815054+0000","last_undegraded":"2024-01-27T03:43:30.815054+0000","last_fullsized":"2024-01-27T03:43:30.815054+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:30.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:32.220723+0000","last_change":"2024-01-27T03:43:30.100170+0000","last_active":"2024-01-27T03:43:32.220723+0000","last_peered":"2024-01-27T03:43:32.220723+0000","last_clean":"2024-01-27T03:43:32.220723+0000","last_became_active":"2024-01-27T03:43:30.072010+0000","last_became_peered":"2024-01-27T03:43:30.072010+0000","last_unstale":"2024-01-27T03:43:32.220723+0000","last_undegraded":"2024-01-27T03:43:32.220723+0000","last_fullsized":"2024-01-27T03:43:32.220723+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:17:46.003541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.805981+0000","last_change":"2024-01-27T03:43:30.123281+0000","last_active":"2024-01-27T03:43:30.805981+0000","last_peered":"2024-01-27T03:43:30.805981+0000","last_clean":"2024-01-27T03:43:30.805981+0000","last_became_active":"2024-01-27T03:43:30.033663+0000","last_became_peered":"2024-01-27T03:43:30.033663+0000","last_unstale":"2024-01-27T03:43:30.805981+0000","last_undegraded":"2024-01-27T03:43:30.805981+0000","last_fullsized":"2024-01-27T03:43:30.805981+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:03.978730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.812203+0000","last_change":"2024-01-27T03:43:30.313647+0000","last_active":"2024-01-27T03:43:30.812203+0000","last_peered":"2024-01-27T03:43:30.812203+0000","last_clean":"2024-01-27T03:43:30.812203+0000","last_became_active":"2024-01-27T03:43:30.303196+0000","last_became_peered":"2024-01-27T03:43:30.303196+0000","last_unstale":"2024-01-27T03:43:30.812203+0000","last_undegraded":"2024-01-27T03:43:30.812203+0000","last_fullsized":"2024-01-27T03:43:30.812203+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:45:18.950474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:31.843822+0000","last_change":"2024-01-27T03:43:25.035922+0000","last_active":"2024-01-27T03:43:31.843822+0000","last_peered":"2024-01-27T03:43:31.843822+0000","last_clean":"2024-01-27T03:43:31.843822+0000","last_became_active":"2024-01-27T03:43:24.404196+0000","last_became_peered":"2024-01-27T03:43:24.404196+0000","last_unstale":"2024-01-27T03:43:31.843822+0000","last_undegraded":"2024-01-27T03:43:31.843822+0000","last_fullsized":"2024-01-27T03:43:31.843822+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_clean_scrub_stamp":"2024-01-27T03:43:13.902947+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:56:03.692684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":651264,"data_stored":485124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.991}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.029}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.903}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.859}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.367999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.050999999999998}]}]},{"osd":2,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.484}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.030000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3379999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":32.933999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.084}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.620999999999999}]}]},{"osd":6,"up_from":20,"seq":85899345931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0169999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.869999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.931999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.226000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0779999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.064}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.280000000000001}]}]},{"osd":4,"up_from":20,"seq":85899345929,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":81536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.169}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":72.620000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.189}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.383000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.298999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":35.789999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.748000000000001}]}]},{"osd":3,"up_from":16,"seq":68719476744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":374.44799999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.405999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.023}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.074999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":65.329999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":38.396999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":49.798999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4690000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7110000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.617000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6760000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.396000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.467000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.247}]}]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":485124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1980000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2260000000000009}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.167999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9559999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.718999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.350999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.797999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":81536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.185}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.545}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8900000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.5999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.523}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.414999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.436}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T03:43:36.305 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T03:43:36.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.405+0000 7fcabb8a9640 10 monclient: get_monmap_and_config 2024-01-27T03:43:36.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.405+0000 7fcabb8a9640 10 monclient: build_initial_monmap 2024-01-27T03:43:36.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.405+0000 7fcabb8a9640 10 monclient: monmap: 2024-01-27T03:43:36.405 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:36.405 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:36.405 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:36.405643+0000 2024-01-27T03:43:36.406 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:36.405643+0000 2024-01-27T03:43:36.406 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:36.406 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:36.406 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:36.406 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:36.406 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:36.406 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:36.406 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:36.406 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 1 Processor -- start 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 1 -- start start 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 10 monclient: init 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcab40b4680 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4a60 0x7fcab40b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 10 monclient(hunting): picked mon.noname-b con 0x7fcab40b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcab40b3b30 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40afbe0 con 0x7fcab40b3b30 2024-01-27T03:43:36.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40b0890 con 0x7fcab40b4680 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab4142240 con 0x7fcab40b4a60 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.406+0000 7fcabb8a9640 10 monclient(hunting): authenticate will time out at 1499.759828s 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.407+0000 7fcab9e46640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcab40b3b30 0x7fcab40b19b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55520/0 (socket says 172.21.15.110:55520) 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.407+0000 7fcab9e46640 1 -- 172.21.15.110:0/21498784 learned_addr learned my addr 172.21.15.110:0/21498784 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.407+0000 7fcab8e44640 1 --2- 172.21.15.110:0/21498784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4a60 0x7fcab40b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.407+0000 7fcab8e44640 10 monclient(hunting): get_auth_request con 0x7fcab40b4a60 auth_method 0 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.407+0000 7fcab8e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.407+0000 7fcab8e44640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.407+0000 7fcab8e44640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.408+0000 7fcab8e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.408+0000 7fcab8e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:36.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.408+0000 7fcab8e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:36.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.410+0000 7fcab8e44640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-01-27T03:43:36.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.410+0000 7fcab8e44640 1 -- 172.21.15.110:0/21498784 >> v1:172.21.15.162:6789/0 conn(0x7fcab40b4680 legacy=0x7fcab4142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.410+0000 7fcab8e44640 1 -- 172.21.15.110:0/21498784 >> v1:172.21.15.110:6789/0 conn(0x7fcab40b3b30 legacy=0x7fcab40b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.410+0000 7fcab8e44640 10 monclient: _finish_hunting 0 2024-01-27T03:43:36.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.410+0000 7fcab8e44640 1 monclient: found mon.noname-b 2024-01-27T03:43:36.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.410+0000 7fcab8e44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:36.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.410+0000 7fcab8e44640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:36.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.410+0000 7fcab8e44640 1 -- 172.21.15.110:0/21498784 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40b0b00 con 0x7fcab40b4a60 2024-01-27T03:43:36.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.413+0000 7fcab8e44640 1 --2- 172.21.15.110:0/21498784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4a60 0x7fcab40b1470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40097d0 tx=0x7fcaa402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efe74ed036350057 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 1 -- 172.21.15.110:0/21498784 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa402d040 con 0x7fcab40b4a60 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 10 monclient: dump: 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:36.419 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 10 monclient: _finish_auth 0 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 10 monclient: _check_auth_tickets 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 1 -- 172.21.15.110:0/21498784 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa4031810 con 0x7fcab40b4a60 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 1 -- 172.21.15.110:0/21498784 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa4031ba0 con 0x7fcab40b4a60 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcabb8a9640 5 monclient: authenticate success, global_id 4331 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcabb8a9640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcabb8a9640 4 set_mon_vals no callback set 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcaabfff640 10 monclient: dump: 2024-01-27T03:43:36.420 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:36.421 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:36.421 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.421 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.421 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:36.421 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:36.421 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:36.421 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:36.421 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:36.421 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:36.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcabb8a9640 10 monclient: shutdown 2024-01-27T03:43:36.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcabb8a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:36.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcabb8a9640 1 -- 172.21.15.110:0/21498784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4a60 msgr2=0x7fcab40b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.419+0000 7fcabb8a9640 1 --2- 172.21.15.110:0/21498784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4a60 0x7fcab40b1470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40097d0 tx=0x7fcaa402b8b0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.420+0000 7fcabb8a9640 1 -- 172.21.15.110:0/21498784 shutdown_connections 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.420+0000 7fcabb8a9640 1 --2- 172.21.15.110:0/21498784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4a60 0x7fcab40b1470 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.420+0000 7fcabb8a9640 1 -- 172.21.15.110:0/21498784 >> 172.21.15.110:0/21498784 conn(0x7fcab4066490 msgr2=0x7fcab40668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.420+0000 7fcabb8a9640 1 -- 172.21.15.110:0/21498784 shutdown_connections 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.420+0000 7fcabb8a9640 1 -- 172.21.15.110:0/21498784 wait complete. 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.420+0000 7fcabb8a9640 10 monclient: build_initial_monmap 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.420+0000 7fcabb8a9640 10 monclient: monmap: 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:36.425 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 librados: starting msgr at 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 librados: starting objecter 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 Processor -- start 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 -- start start 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 librados: setting wanted keys 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 librados: calling monclient init 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient: init 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:36.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcab40b3b30 0x7fcab413d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient(hunting): picked mon.noname-a con 0x7fcab40b3b30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4680 0x7fcab40aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient(hunting): picked mon.noname-c con 0x7fcab40b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab40a97a0 0x7fcab40ac290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient(hunting): picked mon.noname-b con 0x7fcab40a97a0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab4136b60 con 0x7fcab40b3b30 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab413def0 con 0x7fcab40a97a0 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab40aa8f0 con 0x7fcab40b4680 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.421+0000 7fcabb8a9640 10 monclient(hunting): authenticate will time out at 1499.774746s 2024-01-27T03:43:36.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab8e44640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4680 0x7fcab40aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9e46640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab40a97a0 0x7fcab40ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9645640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcab40b3b30 0x7fcab413d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab8e44640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4680 0x7fcab40aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:38692/0 (socket says 172.21.15.110:38692) 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab8e44640 1 -- 172.21.15.110:0/1789057501 learned_addr learned my addr 172.21.15.110:0/1789057501 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab8e44640 10 monclient(hunting): get_auth_request con 0x7fcab40b4680 auth_method 0 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab8e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab8e44640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab8e44640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9e46640 10 monclient(hunting): get_auth_request con 0x7fcab40a97a0 auth_method 0 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9e46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9e46640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9e46640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9645640 10 monclient(hunting): get_auth_request con 0x7fcab40b3b30 auth_method 0 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9645640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:36.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9645640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.422+0000 7fcab9645640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.423+0000 7fcab9e46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.423+0000 7fcab9e46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.423+0000 7fcab9e46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.423+0000 7fcab8e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.423+0000 7fcab8e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.423+0000 7fcab8e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.423+0000 7fcab9645640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.423+0000 7fcab9645640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:36.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.423+0000 7fcab9645640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:36.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.425+0000 7fcab9e46640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2024-01-27T03:43:36.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.425+0000 7fcab9e46640 1 -- 172.21.15.110:0/1789057501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4680 msgr2=0x7fcab40aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.425+0000 7fcab9e46640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4680 0x7fcab40aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.426+0000 7fcab9e46640 1 -- 172.21.15.110:0/1789057501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcab40b3b30 msgr2=0x7fcab413d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.426+0000 7fcab9e46640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcab40b3b30 0x7fcab413d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.426+0000 7fcab9e46640 10 monclient: _finish_hunting 0 2024-01-27T03:43:36.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.426+0000 7fcab9e46640 1 monclient: found mon.noname-b 2024-01-27T03:43:36.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.426+0000 7fcab9e46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:36.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.426+0000 7fcab9e46640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:36.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.426+0000 7fcab9e46640 1 -- 172.21.15.110:0/1789057501 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4009400 con 0x7fcab40a97a0 2024-01-27T03:43:36.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.428+0000 7fcab9e46640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab40a97a0 0x7fcab40ac290 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00cb00 tx=0x7fcaac002180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:36.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.428+0000 7fcaa9ffb640 10 client.?.objecter ms_handle_connect 0x7fcab40a97a0 2024-01-27T03:43:36.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.428+0000 7fcaa9ffb640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:36.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 1 -- 172.21.15.110:0/1789057501 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac00abd0 con 0x7fcab40a97a0 2024-01-27T03:43:36.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:36.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:36.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 10 monclient: dump: 2024-01-27T03:43:36.432 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:36.432 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:36.432 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.432 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.432 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 10 monclient: _finish_auth 0 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 10 monclient: _check_auth_tickets 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 1 -- 172.21.15.110:0/1789057501 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac00ad30 con 0x7fcab40a97a0 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcabb8a9640 5 monclient: authenticate success, global_id 4378 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcabb8a9640 10 monclient: _renew_subs 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcabb8a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab40abb20 con 0x7fcab40a97a0 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:36.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 1 -- 172.21.15.110:0/1789057501 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac00aea0 con 0x7fcab40a97a0 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaa9ffb640 10 monclient: dump: 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.431+0000 7fcaab7fe640 4 set_mon_vals no callback set 2024-01-27T03:43:36.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.432+0000 7fcabb8a9640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:36.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.432+0000 7fcabb8a9640 10 monclient: _renew_subs 2024-01-27T03:43:36.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.432+0000 7fcabb8a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:36.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.432+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab40aab60 con 0x7fcab40a97a0 2024-01-27T03:43:36.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.432+0000 7fcabb8a9640 1 librados: init done 2024-01-27T03:43:36.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.432+0000 7fcabb8a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:36.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.432+0000 7fcabb8a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:36.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.432+0000 7fcabb8a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:36.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.432+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab4074d10 con 0x7fcab40a97a0 2024-01-27T03:43:36.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.680+0000 7fcaa9ffb640 1 -- 172.21.15.110:0/1789057501 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fcaac007070 con 0x7fcab40a97a0 2024-01-27T03:43:36.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.680+0000 7fcaa9ffb640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fca9806c9a0 0x7fca9806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:36.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.681+0000 7fcab9645640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fca9806c9a0 0x7fca9806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:36.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.681+0000 7fcab9645640 10 monclient: get_auth_request con 0x7fca9806c9a0 auth_method 0 2024-01-27T03:43:36.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.681+0000 7fcaa9ffb640 10 client.4378.objecter ms_handle_connect 0x7fca9806c9a0 2024-01-27T03:43:36.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.681+0000 7fcab9645640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fca9806c9a0 0x7fca9806ee60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcab414dad0 tx=0x7fcab0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:36.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.682+0000 7fcaa9ffb640 1 -- 172.21.15.110:0/1789057501 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcaac0585d0 con 0x7fcab40a97a0 2024-01-27T03:43:36.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.682+0000 7fcaa9ffb640 10 client.4378.objecter ms_dispatch 0x7fcab4000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:36.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.682+0000 7fcaa9ffb640 3 client.4378.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:36.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.682+0000 7fcaa9ffb640 3 client.4378.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:36.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.682+0000 7fcaa9ffb640 20 client.4378.objecter dump_active .. 0 homeless 2024-01-27T03:43:36.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.694+0000 7fcaa9ffb640 1 -- 172.21.15.110:0/1789057501 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fcaac054870 con 0x7fcab40a97a0 2024-01-27T03:43:36.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.694+0000 7fcaa9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:36.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.694+0000 7fcaa9ffb640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:36.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.870+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcab40af760 con 0x7fca9806c9a0 2024-01-27T03:43:36.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.872+0000 7fcaa9ffb640 1 -- 172.21.15.110:0/1789057501 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48006 (secure 0 0 0) 0x7fcab40af760 con 0x7fca9806c9a0 2024-01-27T03:43:36.873 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:43:36.873 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-27T03:43:36.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 10 librados: watch_flush enter 2024-01-27T03:43:36.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 10 librados: watch_flush exit 2024-01-27T03:43:36.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 20 client.4378.objecter shutdown clearing up homeless session... 2024-01-27T03:43:36.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 10 client.4378.objecter successfully canceled tick 2024-01-27T03:43:36.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fca9806c9a0 msgr2=0x7fca9806ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fca9806c9a0 0x7fca9806ee60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcab414dad0 tx=0x7fcab0009040 comp rx=0 tx=0).stop 2024-01-27T03:43:36.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 10 monclient: shutdown 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab40a97a0 msgr2=0x7fcab40ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab40a97a0 0x7fcab40ac290 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00cb00 tx=0x7fcaac002180 comp rx=0 tx=0).stop 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 shutdown_connections 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fca9806c9a0 0x7fca9806ee60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcab40b4680 0x7fcab40aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab40a97a0 0x7fcab40ac290 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 --2- 172.21.15.110:0/1789057501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcab40b3b30 0x7fcab413d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 >> 172.21.15.110:0/1789057501 conn(0x7fcab4066490 msgr2=0x7fcab40b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 shutdown_connections 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 -- 172.21.15.110:0/1789057501 wait complete. 2024-01-27T03:43:36.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.874+0000 7fcabb8a9640 1 librados: shutdown 2024-01-27T03:43:36.885 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-01-27T03:43:36.843826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218912,"kb_used_data":2736,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755808,"statfs":{"total":773094113280,"available":772869947392,"internally_reserved":0,"allocated":2801664,"data_stored":1493952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001512"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.824345+0000","last_change":"2024-01-27T03:43:30.350497+0000","last_active":"2024-01-27T03:43:30.824345+0000","last_peered":"2024-01-27T03:43:30.824345+0000","last_clean":"2024-01-27T03:43:30.824345+0000","last_became_active":"2024-01-27T03:43:30.345375+0000","last_became_peered":"2024-01-27T03:43:30.345375+0000","last_unstale":"2024-01-27T03:43:30.824345+0000","last_undegraded":"2024-01-27T03:43:30.824345+0000","last_fullsized":"2024-01-27T03:43:30.824345+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:58.675410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:32.870140+0000","last_change":"2024-01-27T03:43:30.372081+0000","last_active":"2024-01-27T03:43:32.870140+0000","last_peered":"2024-01-27T03:43:32.870140+0000","last_clean":"2024-01-27T03:43:32.870140+0000","last_became_active":"2024-01-27T03:43:30.335129+0000","last_became_peered":"2024-01-27T03:43:30.335129+0000","last_unstale":"2024-01-27T03:43:32.870140+0000","last_undegraded":"2024-01-27T03:43:32.870140+0000","last_fullsized":"2024-01-27T03:43:32.870140+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:01.568710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.818107+0000","last_change":"2024-01-27T03:43:30.077908+0000","last_active":"2024-01-27T03:43:30.818107+0000","last_peered":"2024-01-27T03:43:30.818107+0000","last_clean":"2024-01-27T03:43:30.818107+0000","last_became_active":"2024-01-27T03:43:30.057792+0000","last_became_peered":"2024-01-27T03:43:30.057792+0000","last_unstale":"2024-01-27T03:43:30.818107+0000","last_undegraded":"2024-01-27T03:43:30.818107+0000","last_fullsized":"2024-01-27T03:43:30.818107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:05.446156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.734027+0000","last_change":"2024-01-27T03:43:30.340472+0000","last_active":"2024-01-27T03:43:30.734027+0000","last_peered":"2024-01-27T03:43:30.734027+0000","last_clean":"2024-01-27T03:43:30.734027+0000","last_became_active":"2024-01-27T03:43:30.329560+0000","last_became_peered":"2024-01-27T03:43:30.329560+0000","last_unstale":"2024-01-27T03:43:30.734027+0000","last_undegraded":"2024-01-27T03:43:30.734027+0000","last_fullsized":"2024-01-27T03:43:30.734027+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:10:58.217001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.815054+0000","last_change":"2024-01-27T03:43:30.335032+0000","last_active":"2024-01-27T03:43:30.815054+0000","last_peered":"2024-01-27T03:43:30.815054+0000","last_clean":"2024-01-27T03:43:30.815054+0000","last_became_active":"2024-01-27T03:43:30.329967+0000","last_became_peered":"2024-01-27T03:43:30.329967+0000","last_unstale":"2024-01-27T03:43:30.815054+0000","last_undegraded":"2024-01-27T03:43:30.815054+0000","last_fullsized":"2024-01-27T03:43:30.815054+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:30.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:32.220723+0000","last_change":"2024-01-27T03:43:30.100170+0000","last_active":"2024-01-27T03:43:32.220723+0000","last_peered":"2024-01-27T03:43:32.220723+0000","last_clean":"2024-01-27T03:43:32.220723+0000","last_became_active":"2024-01-27T03:43:30.072010+0000","last_became_peered":"2024-01-27T03:43:30.072010+0000","last_unstale":"2024-01-27T03:43:32.220723+0000","last_undegraded":"2024-01-27T03:43:32.220723+0000","last_fullsized":"2024-01-27T03:43:32.220723+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:17:46.003541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.805981+0000","last_change":"2024-01-27T03:43:30.123281+0000","last_active":"2024-01-27T03:43:30.805981+0000","last_peered":"2024-01-27T03:43:30.805981+0000","last_clean":"2024-01-27T03:43:30.805981+0000","last_became_active":"2024-01-27T03:43:30.033663+0000","last_became_peered":"2024-01-27T03:43:30.033663+0000","last_unstale":"2024-01-27T03:43:30.805981+0000","last_undegraded":"2024-01-27T03:43:30.805981+0000","last_fullsized":"2024-01-27T03:43:30.805981+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:03.978730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:30.812203+0000","last_change":"2024-01-27T03:43:30.313647+0000","last_active":"2024-01-27T03:43:30.812203+0000","last_peered":"2024-01-27T03:43:30.812203+0000","last_clean":"2024-01-27T03:43:30.812203+0000","last_became_active":"2024-01-27T03:43:30.303196+0000","last_became_peered":"2024-01-27T03:43:30.303196+0000","last_unstale":"2024-01-27T03:43:30.812203+0000","last_undegraded":"2024-01-27T03:43:30.812203+0000","last_fullsized":"2024-01-27T03:43:30.812203+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:45:18.950474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-27T03:43:31.843822+0000","last_change":"2024-01-27T03:43:25.035922+0000","last_active":"2024-01-27T03:43:31.843822+0000","last_peered":"2024-01-27T03:43:31.843822+0000","last_clean":"2024-01-27T03:43:31.843822+0000","last_became_active":"2024-01-27T03:43:24.404196+0000","last_became_peered":"2024-01-27T03:43:24.404196+0000","last_unstale":"2024-01-27T03:43:31.843822+0000","last_undegraded":"2024-01-27T03:43:31.843822+0000","last_fullsized":"2024-01-27T03:43:31.843822+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_clean_scrub_stamp":"2024-01-27T03:43:13.902947+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:56:03.692684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":651264,"data_stored":485124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.991}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.029}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.903}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.859}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.367999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.050999999999998}]}]},{"osd":2,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.484}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.030000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.625}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3379999999999992}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":32.933999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.084}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.620999999999999}]}]},{"osd":6,"up_from":20,"seq":85899345932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.025}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.468}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2119999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.277000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.07}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.292999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.644}]}]},{"osd":4,"up_from":20,"seq":85899345930,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1929999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.619}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.733000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.089}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.402000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.476000000000001}]}]},{"osd":3,"up_from":16,"seq":68719476744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":374.44799999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.405999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.023}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.074999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":65.329999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":38.396999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":49.798999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4690000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7110000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.617000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6760000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.396000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.467000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.247}]}]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":485124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1980000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2260000000000009}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.167999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9559999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.718999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.350999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.797999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":87284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9329999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.329000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.27}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.927}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3390000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.183}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T03:43:36.887 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-27T03:43:36.887 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-27T03:43:36.887 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-27T03:43:36.887 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-27T03:43:36.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.987+0000 7fe18578a640 10 monclient: get_monmap_and_config 2024-01-27T03:43:36.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.987+0000 7fe18578a640 10 monclient: build_initial_monmap 2024-01-27T03:43:36.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.987+0000 7fe18578a640 10 monclient: monmap: 2024-01-27T03:43:36.987 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:36.987 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:36.987 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:43:36.987637+0000 2024-01-27T03:43:36.988 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:43:36.987637+0000 2024-01-27T03:43:36.988 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:36.988 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:36.988 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T03:43:36.988 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T03:43:36.988 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T03:43:36.988 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T03:43:36.988 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T03:43:36.988 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:36.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 1 Processor -- start 2024-01-27T03:43:36.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 1 -- start start 2024-01-27T03:43:36.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 10 monclient: init 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1800b4b00 0x7fe1800b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 10 monclient(hunting): picked mon.noname-b con 0x7fe1800b4b00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe18013f020 addr v1:172.21.15.110:6789/0 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe180142060 addr v1:172.21.15.162:6789/0 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1800aaf30 con 0x7fe18013f020 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1800ab210 con 0x7fe180142060 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800b44d0 con 0x7fe1800b4b00 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.988+0000 7fe18578a640 10 monclient(hunting): authenticate will time out at 1500.341877s 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.989+0000 7fe177fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe18013f020 0x7fe18013f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55530/0 (socket says 172.21.15.110:55530) 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.989+0000 7fe177fff640 1 -- 172.21.15.110:0/3963606238 learned_addr learned my addr 172.21.15.110:0/3963606238 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T03:43:36.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.989+0000 7fe17effd640 1 --2- 172.21.15.110:0/3963606238 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1800b4b00 0x7fe1800b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:36.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.989+0000 7fe17effd640 10 monclient(hunting): get_auth_request con 0x7fe1800b4b00 auth_method 0 2024-01-27T03:43:36.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.989+0000 7fe17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:36.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.989+0000 7fe17effd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:36.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.989+0000 7fe17effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:36.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.990+0000 7fe17effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:36.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.990+0000 7fe17effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:36.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.990+0000 7fe17effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:36.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.991+0000 7fe17e7fc640 1 -- 172.21.15.110:0/3963606238 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055188587 0 0) 0x7fe1800ab210 con 0x7fe180142060 2024-01-27T03:43:36.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.991+0000 7fe17e7fc640 10 monclient(hunting): _negotiate 2024-01-27T03:43:36.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.991+0000 7fe17e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:36.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.991+0000 7fe17e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:36.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17e7fc640 10 monclient(hunting): my global_id is 4381 2024-01-27T03:43:36.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17e7fc640 1 -- 172.21.15.110:0/3963606238 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c003520 con 0x7fe180142060 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17effd640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17e7fc640 1 -- 172.21.15.110:0/3963606238 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913811065 0 0) 0x7fe1800aaf30 con 0x7fe18013f020 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17effd640 1 -- 172.21.15.110:0/3963606238 >> v1:172.21.15.162:6789/0 conn(0x7fe180142060 legacy=0x7fe1801424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17effd640 1 -- 172.21.15.110:0/3963606238 >> v1:172.21.15.110:6789/0 conn(0x7fe18013f020 legacy=0x7fe18013f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17effd640 10 monclient: _finish_hunting 0 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17effd640 1 monclient: found mon.noname-b 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17effd640 1 -- 172.21.15.110:0/3963606238 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe180142c00 con 0x7fe1800b4b00 2024-01-27T03:43:36.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.992+0000 7fe17e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-27T03:43:36.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.994+0000 7fe17effd640 1 --2- 172.21.15.110:0/3963606238 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1800b4b00 0x7fe1800b3db0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe1700097d0 tx=0x7fe17002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdf471a7fed4a299 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:36.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 1 -- 172.21.15.110:0/3963606238 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170031770 con 0x7fe1800b4b00 2024-01-27T03:43:36.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:36.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 10 monclient: dump: 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 10 monclient: _finish_auth 0 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:36.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 1 -- 172.21.15.110:0/3963606238 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1700318d0 con 0x7fe1800b4b00 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 1 -- 172.21.15.110:0/3963606238 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170031c60 con 0x7fe1800b4b00 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 5 monclient: authenticate success, global_id 4337 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 10 monclient: get_monmap_and_config success 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 4 set_mon_vals no callback set 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe17e7fc640 10 monclient: dump: 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:37.000 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 10 monclient: shutdown 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 1 -- 172.21.15.110:0/3963606238 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1800b4b00 msgr2=0x7fe1800b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 1 --2- 172.21.15.110:0/3963606238 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1800b4b00 0x7fe1800b3db0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe1700097d0 tx=0x7fe17002b810 comp rx=0 tx=0).stop 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 1 -- 172.21.15.110:0/3963606238 shutdown_connections 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 1 --2- 172.21.15.110:0/3963606238 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1800b4b00 0x7fe1800b3db0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:37.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 1 -- 172.21.15.110:0/3963606238 >> 172.21.15.110:0/3963606238 conn(0x7fe180066490 msgr2=0x7fe1800668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:37.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 1 -- 172.21.15.110:0/3963606238 shutdown_connections 2024-01-27T03:43:37.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.998+0000 7fe18578a640 1 -- 172.21.15.110:0/3963606238 wait complete. 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.999+0000 7fe18578a640 10 monclient: build_initial_monmap 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.999+0000 7fe18578a640 10 monclient: monmap: 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:43:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.999+0000 7fe18578a640 1 librados: starting msgr at 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.999+0000 7fe18578a640 1 librados: starting objecter 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.999+0000 7fe18578a640 1 Processor -- start 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.999+0000 7fe18578a640 1 -- start start 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.999+0000 7fe18578a640 1 librados: setting wanted keys 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.999+0000 7fe18578a640 1 librados: calling monclient init 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:36.999+0000 7fe18578a640 10 monclient: init 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient: _reopen_session rank -1 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1800b4b00 0x7fe180144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient(hunting): picked mon.noname-b con 0x7fe1800b4b00 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe18013f020 0x7fe180137b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient(hunting): picked mon.noname-c con 0x7fe18013f020 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:43:37.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe180142060 0x7fe1801361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient(hunting): picked mon.noname-a con 0x7fe180142060 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1801385a0 con 0x7fe180142060 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe180137340 con 0x7fe1800b4b00 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient(hunting): start opening mon connection 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe180138080 con 0x7fe18013f020 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient(hunting): _renew_subs 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe18578a640 10 monclient(hunting): authenticate will time out at 1500.353350s 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe17f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe180142060 0x7fe1801361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.000+0000 7fe17effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1800b4b00 0x7fe180144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1800b4b00 0x7fe180144f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:40900/0 (socket says 172.21.15.110:40900) 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17effd640 1 -- 172.21.15.110:0/3212232381 learned_addr learned my addr 172.21.15.110:0/3212232381 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17effd640 10 monclient(hunting): get_auth_request con 0x7fe1800b4b00 auth_method 0 2024-01-27T03:43:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17effd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17f7fe640 10 monclient(hunting): get_auth_request con 0x7fe180142060 auth_method 0 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.001+0000 7fe17f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.002+0000 7fe177fff640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe18013f020 0x7fe180137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.002+0000 7fe177fff640 10 monclient(hunting): get_auth_request con 0x7fe18013f020 auth_method 0 2024-01-27T03:43:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.002+0000 7fe177fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.002+0000 7fe177fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.002+0000 7fe177fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.003+0000 7fe177fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.003+0000 7fe177fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.003+0000 7fe177fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 1 -- 172.21.15.110:0/3212232381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe18013f020 msgr2=0x7fe180137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe18013f020 0x7fe180137b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 1 -- 172.21.15.110:0/3212232381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe180142060 msgr2=0x7fe1801361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe180142060 0x7fe1801361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 10 monclient: _finish_hunting 0 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 1 monclient: found mon.noname-b 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:43:37.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.004+0000 7fe17effd640 1 -- 172.21.15.110:0/3212232381 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe170009400 con 0x7fe1800b4b00 2024-01-27T03:43:37.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.006+0000 7fe17effd640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1800b4b00 0x7fe180144f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe1700097d0 tx=0x7fe1700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:37.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.006+0000 7fe1777fe640 10 client.?.objecter ms_handle_connect 0x7fe1800b4b00 2024-01-27T03:43:37.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.006+0000 7fe1777fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:43:37.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 1 -- 172.21.15.110:0/3212232381 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170004230 con 0x7fe1800b4b00 2024-01-27T03:43:37.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:37.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 10 monclient: dump: 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 10 monclient: _finish_auth 0 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 10 monclient: _check_auth_tickets 2024-01-27T03:43:37.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 1 -- 172.21.15.110:0/3212232381 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1700043b0 con 0x7fe1800b4b00 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe18578a640 5 monclient: authenticate success, global_id 4384 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe18578a640 10 monclient: _renew_subs 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe18578a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1801366e0 con 0x7fe1800b4b00 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 1 -- 172.21.15.110:0/3212232381 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170004520 con 0x7fe1800b4b00 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe1777fe640 10 monclient: dump: 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T03:43:37.013 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe17dffb640 4 set_mon_vals no callback set 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe18578a640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe18578a640 10 monclient: _renew_subs 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe18578a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe180136bf0 con 0x7fe1800b4b00 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.010+0000 7fe18578a640 1 librados: init done 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.011+0000 7fe18578a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.011+0000 7fe18578a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.011+0000 7fe18578a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:37.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.011+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18004ea50 con 0x7fe1800b4b00 2024-01-27T03:43:37.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.252+0000 7fe1777fe640 1 -- 172.21.15.110:0/3212232381 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fe170043070 con 0x7fe1800b4b00 2024-01-27T03:43:37.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.252+0000 7fe1777fe640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe16006c9f0 0x7fe16006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:43:37.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.253+0000 7fe177fff640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe16006c9f0 0x7fe16006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:43:37.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.253+0000 7fe177fff640 10 monclient: get_auth_request con 0x7fe16006c9f0 auth_method 0 2024-01-27T03:43:37.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.253+0000 7fe1777fe640 10 client.4384.objecter ms_handle_connect 0x7fe16006c9f0 2024-01-27T03:43:37.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.253+0000 7fe177fff640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe16006c9f0 0x7fe16006eeb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe168007f70 tx=0x7fe168007f00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:43:37.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.256+0000 7fe1777fe640 1 -- 172.21.15.110:0/3212232381 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe17007a000 con 0x7fe1800b4b00 2024-01-27T03:43:37.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.256+0000 7fe1777fe640 10 client.4384.objecter ms_dispatch 0x7fe180000b90 osd_map(26..26 src has 1..26) v4 2024-01-27T03:43:37.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.256+0000 7fe1777fe640 3 client.4384.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:43:37.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.256+0000 7fe1777fe640 3 client.4384.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:43:37.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.256+0000 7fe1777fe640 20 client.4384.objecter dump_active .. 0 homeless 2024-01-27T03:43:37.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.267+0000 7fe1777fe640 1 -- 172.21.15.110:0/3212232381 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fe1700762a0 con 0x7fe1800b4b00 2024-01-27T03:43:37.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.267+0000 7fe1777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T03:43:37.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.267+0000 7fe1777fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T03:43:37.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.509+0000 7fe18578a640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-01-27T03:43:37.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.509+0000 7fe18578a640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-01-27T03:43:37.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.509+0000 7fe18578a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:43:37.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.509+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe1801386e0 con 0x7fe1800b4b00 2024-01-27T03:43:37.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.515+0000 7fe1777fe640 1 -- 172.21.15.110:0/3212232381 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe170079960 con 0x7fe1800b4b00 2024-01-27T03:43:37.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.515+0000 7fe1777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-01-27T03:43:37.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.515+0000 7fe1777fe640 10 monclient: _finish_command 2 = system:0 2024-01-27T03:43:37.516 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T03:43:37.516 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-27T03:43:37.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.517+0000 7fe18578a640 10 librados: watch_flush enter 2024-01-27T03:43:37.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.517+0000 7fe18578a640 10 librados: watch_flush exit 2024-01-27T03:43:37.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.517+0000 7fe18578a640 20 client.4384.objecter shutdown clearing up homeless session... 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.517+0000 7fe18578a640 10 client.4384.objecter successfully canceled tick 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe16006c9f0 msgr2=0x7fe16006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe16006c9f0 0x7fe16006eeb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe168007f70 tx=0x7fe168007f00 comp rx=0 tx=0).stop 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 10 monclient: shutdown 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1800b4b00 msgr2=0x7fe180144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1800b4b00 0x7fe180144f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe1700097d0 tx=0x7fe1700046c0 comp rx=0 tx=0).stop 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 shutdown_connections 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe16006c9f0 0x7fe16006eeb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe18013f020 0x7fe180137b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe180142060 0x7fe1801361a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 --2- 172.21.15.110:0/3212232381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1800b4b00 0x7fe180144f00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 >> 172.21.15.110:0/3212232381 conn(0x7fe180066490 msgr2=0x7fe1801424f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 shutdown_connections 2024-01-27T03:43:37.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 -- 172.21.15.110:0/3212232381 wait complete. 2024-01-27T03:43:37.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T03:43:37.518+0000 7fe18578a640 1 librados: shutdown 2024-01-27T03:43:37.528 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-27T03:43:37.529 INFO:teuthology.run_tasks:Running task workunit... 2024-01-27T03:43:37.547 INFO:tasks.workunit:Pulling workunits from ref 5ffe0641f18e6ec692535294010d9456630e7923 2024-01-27T03:43:37.548 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-01-27T03:43:37.548 DEBUG:teuthology.orchestra.run.smithi110:> stat -- /home/ubuntu/cephtest/mnt.0 2024-01-27T03:43:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T03:43:37.564 INFO:teuthology.orchestra.run.smithi110.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-27T03:43:37.564 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-01-27T03:43:37.619 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-01-27T03:43:37.619 DEBUG:teuthology.orchestra.run.smithi110:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-01-27T03:43:37.674 INFO:tasks.workunit:timeout=6h 2024-01-27T03:43:37.674 INFO:tasks.workunit:cleanup=True 2024-01-27T03:43:37.675 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5ffe0641f18e6ec692535294010d9456630e7923 2024-01-27T03:43:37.729 INFO:tasks.workunit.client.0.smithi110.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-01-27T03:43:43.474 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:29.430 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:43.521 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:29.477 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:43.569 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:29.525 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:43.569 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:01:29.515 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:43.572 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:01:29.518 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:43.573 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:29.530 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:43.579 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:29.536 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:43.612 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:01:29.558 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:43:43.618 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:29.574 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:03.570 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:49.526 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:03.574 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:49.531 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:03.590 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:01:49.547 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:03.621 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:49.578 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:03.622 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:01:49.568 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:03.624 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:49.581 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:03.626 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:01:49.571 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:03.638 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:01:49.595 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:03.640 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:01:49.586 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:23.625 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:09.581 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:23.627 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:09.583 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:23.627 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:02:09.573 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:23.650 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:09.606 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:23.652 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:02:09.598 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:23.667 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:02:09.612 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:23.692 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:09.648 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:23.700 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:09.656 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:23.719 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:09.675 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:43.650 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:29.606 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:43.731 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:29.687 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:43.756 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:02:29.712 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:43.778 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:02:29.724 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:43.783 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:02:29.729 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:43.801 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:02:29.747 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:43.805 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:29.761 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:43.808 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:29.765 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:43.834 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:02:29.790 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:44:48.291 INFO:tasks.workunit.client.0.smithi110.stderr:Note: switching to '5ffe0641f18e6ec692535294010d9456630e7923'. 2024-01-27T03:44:48.291 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr:state without impacting any branches by switching back to a branch. 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr: git switch -c 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr:Or undo this operation with: 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr: git switch - 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-27T03:44:48.292 INFO:tasks.workunit.client.0.smithi110.stderr: 2024-01-27T03:44:48.293 INFO:tasks.workunit.client.0.smithi110.stderr:HEAD is now at 5ffe0641f18 osd: clear out unneeded pending pg-upmap-primary mappings 2024-01-27T03:44:48.294 DEBUG:teuthology.orchestra.run.smithi110:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-01-27T03:44:48.357 INFO:tasks.workunit.client.0.smithi110.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-27T03:44:48.359 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-27T03:44:48.359 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-27T03:44:49.025 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-27T03:44:49.091 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-27T03:44:49.130 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-27T03:44:49.132 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-27T03:44:49.132 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-27T03:44:49.170 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-27T03:44:49.173 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-01-27T03:44:49.174 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-01-27T03:44:49.229 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-01-27T03:44:49.230 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-01-27T03:44:49.230 DEBUG:teuthology.orchestra.run.smithi110:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5ffe0641f18e6ec692535294010d9456630e7923 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ parallel=1 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' '' = --serial ']' 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ crimson=0 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' '' = --crimson ']' 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ color= 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -t 1 ']' 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ declare -A pids 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.289 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_aio 2024-01-27T03:44:49.290 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_aio' 2024-01-27T03:44:49.290 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_aio 2024-01-27T03:44:49.290 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.290 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_aio 2024-01-27T03:44:49.290 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40064 2024-01-27T03:44:49.290 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_aio on pid 40064' 2024-01-27T03:44:49.291 INFO:tasks.workunit.client.0.smithi110.stdout:test api_aio on pid 40064 2024-01-27T03:44:49.293 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40064 2024-01-27T03:44:49.293 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.293 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.293 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-01-27T03:44:49.293 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_aio_pp 2024-01-27T03:44:49.293 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_aio_pp' 2024-01-27T03:44:49.293 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_aio_pp 2024-01-27T03:44:49.293 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.293 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.294 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.295 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.295 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.295 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.295 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.295 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.296 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.296 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.296 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.296 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_aio_pp 2024-01-27T03:44:49.296 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40071 2024-01-27T03:44:49.296 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_aio_pp on pid 40071' 2024-01-27T03:44:49.296 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40071 2024-01-27T03:44:49.296 INFO:tasks.workunit.client.0.smithi110.stdout:test api_aio_pp on pid 40071 2024-01-27T03:44:49.300 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.300 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.300 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_io 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_io' 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_io 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.301 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.302 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.303 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.304 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.304 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.304 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.304 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_io 2024-01-27T03:44:49.304 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40080 2024-01-27T03:44:49.304 INFO:tasks.workunit.client.0.smithi110.stdout:test api_io on pid 40080 2024-01-27T03:44:49.308 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_io on pid 40080' 2024-01-27T03:44:49.308 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40080 2024-01-27T03:44:49.308 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-01-27T03:44:49.308 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_io_pp 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_io_pp' 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_io_pp 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.309 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.310 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.311 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.312 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_io_pp 2024-01-27T03:44:49.313 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40092 2024-01-27T03:44:49.313 INFO:tasks.workunit.client.0.smithi110.stdout:test api_io_pp on pid 40092 2024-01-27T03:44:49.318 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_io_pp on pid 40092' 2024-01-27T03:44:49.318 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40092 2024-01-27T03:44:49.318 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.318 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.318 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-01-27T03:44:49.318 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_asio 2024-01-27T03:44:49.318 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_asio' 2024-01-27T03:44:49.318 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_asio 2024-01-27T03:44:49.318 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.319 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.320 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.321 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.322 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.323 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_asio 2024-01-27T03:44:49.324 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.324 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40107 2024-01-27T03:44:49.324 INFO:tasks.workunit.client.0.smithi110.stdout:test api_asio on pid 40107 2024-01-27T03:44:49.330 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_asio on pid 40107' 2024-01-27T03:44:49.330 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40107 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_list 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_list' 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_list 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.331 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.332 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.333 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.334 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.335 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.336 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40064/exe 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.337 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.338 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.338 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_list 2024-01-27T03:44:49.338 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.338 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.338 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40123 2024-01-27T03:44:49.338 INFO:tasks.workunit.client.0.smithi110.stdout:test api_list on pid 40123 2024-01-27T03:44:49.345 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-01-27T03:44:49.345 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_list on pid 40123' 2024-01-27T03:44:49.345 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40123 2024-01-27T03:44:49.345 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.345 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.345 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.345 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_lock 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_lock' 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_lock 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.346 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.347 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.348 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.349 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.350 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_aio: /' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_aio.log 2024-01-27T03:44:49.351 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.352 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40071/exe 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_lock 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40144 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_lock on pid 40144' 2024-01-27T03:44:49.353 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-01-27T03:44:49.354 INFO:tasks.workunit.client.0.smithi110.stdout:test api_lock on pid 40144 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40144 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_lock_pp 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_lock_pp' 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_lock_pp 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.361 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.362 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.363 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.364 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.365 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.366 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.367 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40080/exe 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_aio_pp: /' 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.368 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.369 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_lock_pp 2024-01-27T03:44:49.369 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.369 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40165 2024-01-27T03:44:49.369 INFO:tasks.workunit.client.0.smithi110.stdout:test api_lock_pp on pid 40165 2024-01-27T03:44:49.380 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_lock_pp on pid 40165' 2024-01-27T03:44:49.380 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40165 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_misc 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.381 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.382 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.383 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.384 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_misc' 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_io: /' 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.385 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.386 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.387 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_io.log 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_misc 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40092/exe 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.388 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.389 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.390 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.391 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_misc 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40192 2024-01-27T03:44:49.392 INFO:tasks.workunit.client.0.smithi110.stdout:test api_misc on pid 40192 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_misc on pid 40192' 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40192 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.412 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.413 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_misc_pp 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_misc_pp' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_io_pp: /' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.414 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_misc_pp 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.415 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.416 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.417 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.418 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.419 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.420 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.421 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.422 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.423 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40165/exe 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.424 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.425 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-01-27T03:44:49.426 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_lock_pp: /' 2024-01-27T03:44:49.427 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_misc_pp 2024-01-27T03:44:49.427 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40225 2024-01-27T03:44:49.427 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_misc_pp on pid 40225' 2024-01-27T03:44:49.427 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.427 INFO:tasks.workunit.client.0.smithi110.stdout:test api_misc_pp on pid 40225 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40225 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.442 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_tier_pp 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_tier_pp' 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40123/exe 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.443 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.444 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.445 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_list.log 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.446 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40107/exe 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.447 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40192/exe 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.448 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.449 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40144/exe 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_asio.log 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.450 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.451 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_lock: /' 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_asio: /' 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_lock.log 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_tier_pp 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_list: /' 2024-01-27T03:44:49.452 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_misc.log 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_misc: /' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.453 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.454 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.454 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.454 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_tier_pp 2024-01-27T03:44:49.454 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40284 2024-01-27T03:44:49.454 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_tier_pp on pid 40284' 2024-01-27T03:44:49.454 INFO:tasks.workunit.client.0.smithi110.stdout:test api_tier_pp on pid 40284 2024-01-27T03:44:49.455 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40284 2024-01-27T03:44:49.455 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.455 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.455 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_pool 2024-01-27T03:44:49.455 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_pool' 2024-01-27T03:44:49.456 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_pool 2024-01-27T03:44:49.456 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.456 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_pool 2024-01-27T03:44:49.456 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40290 2024-01-27T03:44:49.456 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_pool on pid 40290' 2024-01-27T03:44:49.456 INFO:tasks.workunit.client.0.smithi110.stdout:test api_pool on pid 40290 2024-01-27T03:44:49.466 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40290 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.467 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_snapshots 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_snapshots' 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.468 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.469 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.470 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_snapshots 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.471 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.472 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.473 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40225/exe 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_snapshots 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40358 2024-01-27T03:44:49.474 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_snapshots on pid 40358' 2024-01-27T03:44:49.475 INFO:tasks.workunit.client.0.smithi110.stdout:test api_snapshots on pid 40358 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40358 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_snapshots_pp 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_snapshots_pp' 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.482 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_snapshots_pp 2024-01-27T03:44:49.483 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.484 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_misc_pp: /' 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.485 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.486 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_snapshots_pp 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40374 2024-01-27T03:44:49.487 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_snapshots_pp on pid 40374' 2024-01-27T03:44:49.488 INFO:tasks.workunit.client.0.smithi110.stdout:test api_snapshots_pp on pid 40374 2024-01-27T03:44:49.499 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40374 2024-01-27T03:44:49.499 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.499 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_stat 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_stat' 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.500 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.501 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.502 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.503 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.504 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40290/exe 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.505 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.506 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.507 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_stat 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_pool.log 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_pool: /' 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_stat 2024-01-27T03:44:49.508 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40413 2024-01-27T03:44:49.509 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.509 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.509 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_stat on pid 40413' 2024-01-27T03:44:49.509 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.509 INFO:tasks.workunit.client.0.smithi110.stdout:test api_stat on pid 40413 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40413 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.515 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.516 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.517 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_stat_pp 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_stat_pp' 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.518 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_stat_pp 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.519 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.520 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_stat_pp 2024-01-27T03:44:49.520 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40431 2024-01-27T03:44:49.520 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.520 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.520 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.520 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_stat_pp on pid 40431' 2024-01-27T03:44:49.520 INFO:tasks.workunit.client.0.smithi110.stdout:test api_stat_pp on pid 40431 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40431 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_watch_notify 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.529 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.530 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_watch_notify' 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.531 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_watch_notify 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.532 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.533 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.534 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_watch_notify 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.535 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.536 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.536 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.536 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40448 2024-01-27T03:44:49.536 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.536 INFO:tasks.workunit.client.0.smithi110.stdout:test api_watch_notify on pid 40448 2024-01-27T03:44:49.548 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.548 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_watch_notify on pid 40448' 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40448 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.549 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40358/exe 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.550 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40284/exe 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.551 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.552 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_watch_notify_pp 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40374/exe 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_watch_notify_pp' 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.553 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_watch_notify_pp 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.554 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_snapshots: /' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.555 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.556 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.557 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_tier_pp: /' 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.558 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.559 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [----------] Global test environment set-up. 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [----------] 12 tests from AsioRados 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.429+0000 7fa0b05fa780 10 monclient: get_monmap_and_config 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.429+0000 7fa0b05fa780 10 monclient: build_initial_monmap 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.429+0000 7fa0b05fa780 10 monclient: monmap: 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: epoch 0 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: last_changed 0.000000 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: created 0.000000 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: min_mon_release 0 (unknown) 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: election_strategy: 1 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:44:49.560 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.429+0000 7fa0b05fa780 1 Processor -- start 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 1 -- start start 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient: init 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient: _reopen_session rank -1 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d1be480 0x55eb8d251da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient(hunting): picked mon.noname-c con 0x55eb8d1be480 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1c1910 0x55eb8d259730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient(hunting): picked mon.noname-b con 0x55eb8d1c1910 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb8d24fb10 0x55eb8d24def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient(hunting): picked mon.noname-a con 0x55eb8d24fb10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient(hunting): start opening mon connection 2024-01-27T03:44:49.561 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb8d13ac10 con 0x55eb8d24fb10 2024-01-27T03:44:49.562 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient(hunting): start opening mon connection 2024-01-27T03:44:49.562 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb8d0f7130 con 0x55eb8d1c1910 2024-01-27T03:44:49.562 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient(hunting): start opening mon connection 2024-01-27T03:44:49.562 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eb8d0f6dd0 con 0x55eb8d1be480 2024-01-27T03:44:49.562 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient(hunting): _renew_subs 2024-01-27T03:44:49.562 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.430+0000 7fa0b05fa780 10 monclient(hunting): authenticate will time out at 1572.784864s 2024-01-27T03:44:49.564 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.564 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.564 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.564 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.564 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.565 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.566 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_watch_notify_pp 2024-01-27T03:44:49.566 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40490 2024-01-27T03:44:49.566 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_watch_notify_pp on pid 40490' 2024-01-27T03:44:49.566 INFO:tasks.workunit.client.0.smithi110.stdout: test api_watch_notify_pp on pid 40490 2024-01-27T03:44:49.578 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40490 2024-01-27T03:44:49.578 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.578 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.579 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.580 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.581 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.582 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.583 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.584 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_cmd 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_cmd' 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40413/exe 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.585 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.586 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.587 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.588 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.441+0000 7fa0ae8ff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb8d24fb10 0x55eb8d24def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.447+0000 7fa0ae8ff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb8d24fb10 0x55eb8d24def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:35038/0 (socket says 172.21.15.110:35038) 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.447+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/476778340 learned_addr learned my addr 172.21.15.110:0/476778340 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.447+0000 7fa0ae8ff640 10 monclient(hunting): get_auth_request con 0x55eb8d24fb10 auth_method 0 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.447+0000 7fa0ae8ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.447+0000 7fa0ae8ff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.447+0000 7fa0ae8ff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:44:49.589 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.455+0000 7fa0ae0fe640 1 --2- 172.21.15.110:0/476778340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d1be480 0x55eb8d251da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.456+0000 7fa0ae0fe640 10 monclient(hunting): get_auth_request con 0x55eb8d1be480 auth_method 0 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.456+0000 7fa0ae0fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.456+0000 7fa0ae0fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.456+0000 7fa0ae0fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.463+0000 7fa0ae0fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.463+0000 7fa0ae0fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.463+0000 7fa0ae0fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.464+0000 7fa0ad8fd640 1 --2- 172.21.15.110:0/476778340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1c1910 0x55eb8d259730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.464+0000 7fa0ad8fd640 10 monclient(hunting): get_auth_request con 0x55eb8d1c1910 auth_method 0 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.466+0000 7fa0ad8fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.466+0000 7fa0ad8fd640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.466+0000 7fa0ad8fd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.467+0000 7fa0ae8ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.467+0000 7fa0ae8ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:44:49.590 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.467+0000 7fa0ae8ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:44:49.591 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.468+0000 7fa0ae0fe640 10 monclient(hunting): handle_auth_done global_id 4373 payload 274 2024-01-27T03:44:49.593 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40431/exe 2024-01-27T03:44:49.593 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.593 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.593 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.593 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_cmd 2024-01-27T03:44:49.593 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.593 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.594 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_stat.log 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_stat: /' 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_cmd 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40533 2024-01-27T03:44:49.595 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.468+0000 7fa0ae0fe640 1 -- 172.21.15.110:0/476778340 >> [v2:172.21.15test api_cmd on pid 40533 2024-01-27T03:44:49.598 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_cmd on pid 40533' 2024-01-27T03:44:49.598 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40533 2024-01-27T03:44:49.598 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.598 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.598 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-01-27T03:44:49.598 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2024-01-27T03:44:49.598 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-01-27T03:44:49.598 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_stat_pp: /' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_cmd_pp 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_cmd_pp' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_cmd_pp 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.599 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.600 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.600 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.600 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.600 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_cmd_pp 2024-01-27T03:44:49.600 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.600 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40549 2024-01-27T03:44:49.600 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_cmd_pp on pid 40549' 2024-01-27T03:44:49.600 INFO:tasks.workunit.client.0.smithi110.stdout:test api_cmd_pp on pid 40549 2024-01-27T03:44:49.606 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40549 2024-01-27T03:44:49.606 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.606 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.606 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.606 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.606 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.606 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.607 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.608 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.609 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stdout:.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1c1910 msgr2=0x55eb8d259730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:44:49.610 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.468+0000 7fa0ae0fe640 1 --2- 172.21.15.110:0/476778340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1c1910 0x55eb8d259730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.468+0000 7fa0ae0fe640 1 -- 172.21.15.110:0/476778340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb8d24fb10 msgr2=0x55eb8d24def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.468+0000 7fa0ae0fe640 1 --2- 172.21.15.110:0/476778340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb8d24fb10 0x55eb8d24def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.469+0000 7fa0ae0fe640 10 monclient: _finish_hunting 0 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.469+0000 7fa0ae0fe640 1 monclient: found mon.noname-c 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.469+0000 7fa0ae0fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.469+0000 7fa0ae0fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.469+0000 7fa0ae0fe640 1 -- 172.21.15.110:0/476778340 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a4005d20 con 0x55eb8d1be480 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.475+0000 7fa0ae0fe640 1 --2- 172.21.15.110:0/476778340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d1be480 0x55eb8d251da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4051000 tx=0x7fa0a404c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b568a8dafeb14be server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 1 -- 172.21.15.110:0/476778340 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4052640 con 0x55eb8d1be480 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 10 monclient: dump: 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: epoch 1 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:44:49.611 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: min_mon_release 19 (squid) 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: election_strategy: 1 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 10 monclient: _finish_auth 0 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 10 monclient: _check_auth_tickets 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 1 -- 172.21.15.110:0/476778340 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a40527c0 con 0x55eb8d1be480 2024-01-27T03:44:49.612 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:44:49.613 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.613 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.613 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.613 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 1 -- 172.21.15.110:0/476778340 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4052b70 con 0x55eb8d1be480 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0ac8fb640 10 monclient: dump: 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: epoch 1 2024-01-27T03:44:49.614 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: min_mon_release 19 (squid) 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: election_strategy: 1 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0b05fa780 5 monclient: authenticate success, global_id 4373 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0b05fa780 10 monclient: get_monmap_and_config success 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.489+0000 7fa0b05fa780 4 set_mon_vals no callback set 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.490+0000 7fa0b05fa780 10 monclient: shutdown 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.490+0000 7fa0b05fa780 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.490+0000 7fa0b05fa780 1 -- 172.21.15.110:0/476778340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d1be480 msgr2=0x55eb8d251da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.490+0000 7fa0b05fa780 1 --2- 172.21.15.110:0/476778340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d1be480 0x55eb8d251da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4051000 tx=0x7fa0a404c000 comp rx=0 tx=0).stop 2024-01-27T03:44:49.615 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.491+0000 7fa0b05fa780 1 -- 172.21.15.110:0/476778340 shutdown_connections 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.491+0000 7fa0b05fa780 1 --2- 172.21.15.110:0/476778340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb8d24fb10 0x55eb8d24def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.491+0000 7fa0b05fa780 1 --2- 172.21.15.110:0/476778340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1c1910 0x55eb8d259730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.491+0000 7fa0b05fa780 1 --2- 172.21.15.110:0/476778340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d1be480 0x55eb8d251da0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.491+0000 7fa0b05fa780 1 -- 172.21.15.110:0/476778340 >> 172.21.15.110:0/476778340 conn(0x55eb8d1c4ac0 msgr2=0x55eb8d1bb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.491+0000 7fa0b05fa780 1 -- 172.21.15.110:0/476778340 shutdown_connections 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.491+0000 7fa0b05fa780 1 -- 172.21.15.110:0/476778340 wait complete. 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.491+0000 7fa0b05fa780 10 monclient: build_initial_monmap 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.491+0000 7fa0b05fa780 10 monclient: monmap: 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: epoch 0 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: last_changed 0.000000 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: created 0.000000 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: min_mon_release 0 (unknown) 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: election_strategy: 1 2024-01-27T03:44:49.616 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.622 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.623 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.624 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40490/exe 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_service 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_service' 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.625 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_service 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_service 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40599 2024-01-27T03:44:49.626 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_service on pid 40599' 2024-01-27T03:44:49.627 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-01-27T03:44:49.627 INFO:tasks.workunit.client.0.smithi110.stdout: api_atest api_service on pid 40599 2024-01-27T03:44:49.628 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40599 2024-01-27T03:44:49.628 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.628 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.628 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_service_pp 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_service_pp' 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stdout:sio: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T03:44:49.629 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.492+0000 7fa0b05fa780 1 librados: starting msgr at 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.492+0000 7fa0b05fa780 1 librados: starting objecter 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.492+0000 7fa0b05fa780 1 Processor -- start 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.492+0000 7fa0b05fa780 1 -- start start 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.492+0000 7fa0b05fa780 1 librados: setting wanted keys 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.492+0000 7fa0b05fa780 1 librados: calling monclient init 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.492+0000 7fa0b05fa780 10 monclient: init 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient: _reopen_session rank -1 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1be480 0x55eb8d258c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient(hunting): picked mon.noname-b con 0x55eb8d1be480 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb8d1c1910 0x55eb8d2eb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient(hunting): picked mon.noname-a con 0x55eb8d1c1910 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d24fb10 0x55eb8d1c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:44:49.630 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient(hunting): picked mon.noname-c con 0x55eb8d24fb10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T03:44:49.631 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient(hunting): start opening mon connection 2024-01-27T03:44:49.631 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb8d0b3550 con 0x55eb8d1c1910 2024-01-27T03:44:49.631 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient(hunting): start opening mon connection 2024-01-27T03:44:49.631 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb8d1852f0 con 0x55eb8d1be480 2024-01-27T03:44:49.631 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient(hunting): start opening mon connection 2024-01-27T03:44:49.631 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eb8d24bb90 con 0x55eb8d24fb10 2024-01-27T03:44:49.631 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient(hunting): _renew_subs 2024-01-27T03:44:49.631 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.493+0000 7fa0b05fa780 10 monclient(hunting): authenticate will time out at 1572.847700s 2024-01-27T03:44:49.631 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.501+0000 7fa0ae8ff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d24fb10 0x55eb8d1c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.636 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.637 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.638 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.639 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.504+0000 7fa0ae8ff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d24fb10 0x55eb8d1c2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:57546/0 (socket says 172.21.15.110:57546) 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.504+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 learned_addr learned my addr 172.21.15.110:0/882326301 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.504+0000 7fa0ae8ff640 10 monclient(hunting): get_auth_request con 0x55eb8d24fb10 auth_method 0 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.504+0000 7fa0ae8ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.504+0000 7fa0ae8ff640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.504+0000 7fa0ae8ff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.505+0000 7fa0ae0fe640 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1be480 0x55eb8d258c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.505+0000 7fa0ae0fe640 10 monclient(hunting): get_auth_request con 0x55eb8d1be480 auth_method 0 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.505+0000 7fa0ae0fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.505+0000 7fa0ae0fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.505+0000 7fa0ae0fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T03:44:49.640 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.506+0000 7fa0ae0fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.506+0000 7fa0ae0fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.506+0000 7fa0ae0fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.508+0000 7fa0ae8ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.508+0000 7fa0ae8ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.508+0000 7fa0ae8ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 1 -- 172.21.15.110:0/882326301 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d24fb10 msgr2=0x55eb8d1c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55eb8d24fb10 0x55eb8d1c2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 1 -- 172.21.15.110:0/882326301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb8d1c1910 msgr2=0x55eb8d2eb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55eb8d1c1910 0x55eb8d2eb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 10 monclient: _finish_hunting 0 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 1 monclient: found mon.noname-b 2024-01-27T03:44:49.641 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:44:49.649 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.650 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.651 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40533/exe 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40448/exe 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.652 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.653 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_watch_notify: /' 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_cmd: /' 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.654 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_service_pp 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.655 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_service_pp 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40645 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_service_pp on pid 40645' 2024-01-27T03:44:49.656 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-01-27T03:44:49.657 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 10 monclienttest api_service_pp on pid 40645 2024-01-27T03:44:49.664 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40645 2024-01-27T03:44:49.664 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_c_write_operations 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_c_write_operations' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.665 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_c_write_operations 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.666 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.667 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.668 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.669 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_cmd.log 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_c_write_operations 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40549/exe 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40680 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_c_write_operations on pid 40680' 2024-01-27T03:44:49.670 INFO:tasks.workunit.client.0.smithi110.stdout:test api_c_write_operations on pid 40680 2024-01-27T03:44:49.676 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40680 2024-01-27T03:44:49.676 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.676 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.676 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-01-27T03:44:49.676 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.676 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.676 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.677 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.678 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.679 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_c_read_operations 2024-01-27T03:44:49.680 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_c_read_operations' 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout:: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.510+0000 7fa0ae0fe640 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb8d0fa130 con 0x55eb8d1be480 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.518+0000 7fa0ae0fe640 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1be480 0x55eb8d258c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4000c00 tx=0x7fa0a4002fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.518+0000 7fa09f7fe640 10 client.?.objecter ms_handle_connect 0x55eb8d1be480 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.518+0000 7fa09f7fe640 10 client.?.objecter resend_mon_ops 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4052d40 con 0x55eb8d1be480 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 10 monclient: dump: 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: epoch 1 2024-01-27T03:44:49.681 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: min_mon_release 19 (squid) 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: election_strategy: 1 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 10 monclient: _finish_auth 0 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 10 monclient: _check_auth_tickets 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a4052600 con 0x55eb8d1be480 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4064670 con 0x55eb8d1be480 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T03:44:49.682 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa09f7fe640 10 monclient: dump: 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: epoch 1 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: created 2024-01-27T03:41:43.032048+0000 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: min_mon_release 19 (squid) 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: election_strategy: 1 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.683 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0b05fa780 5 monclient: authenticate success, global_id 4423 2024-01-27T03:44:49.691 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.691 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.691 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.691 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.691 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_c_read_operations 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.692 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40599/exe 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.693 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.694 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.695 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.696 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_service.log 2024-01-27T03:44:49.697 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_c_read_operations 2024-01-27T03:44:49.698 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40725 2024-01-27T03:44:49.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-test api_c_read_operations on pid 40725 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_c_read_operations on pid 40725' 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40725 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s api_cls_remote_reads 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.704 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' api_cls_remote_reads' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_service: /' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.705 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.706 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.707 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo api_cls_remote_reads 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.708 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.709 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.709 INFO:tasks.workunit.client.0.smithi110.stdout:test api_cls_remote_reads on pid 40753 2024-01-27T03:44:49.714 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.715 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.716 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.717 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=api_cls_remote_reads 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40753 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test api_cls_remote_reads on pid 40753' 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40753 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s list_parallel 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' list_parallel' 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.718 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40645/exe 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.719 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo list_parallel 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.720 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.721 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.722 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=list_parallel 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40772 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test list_parallel on pid 40772' 2024-01-27T03:44:49.723 INFO:tasks.workunit.client.0.smithi110.stdout:test list_parallel on pid 40772 2024-01-27T03:44:49.731 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40772 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.732 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40680/exe 2024-01-27T03:44:49.733 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_service_pp: /' 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s open_pools_parallel 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' open_pools_parallel' 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.734 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo open_pools_parallel 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.735 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.736 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=open_pools_parallel 2024-01-27T03:44:49.737 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40795 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.738 INFO:tasks.workunit.client.0.smithi110.stdout:test open_pools_parallel on pid 40795 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test open_pools_parallel on pid 40795' 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40795 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s delete_pools_parallel 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' delete_pools_parallel' 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.742 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo delete_pools_parallel 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.743 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=delete_pools_parallel 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.744 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40821 2024-01-27T03:44:49.745 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.745 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.745 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test delete_pools_parallel on pid 40821' 2024-01-27T03:44:49.745 INFO:tasks.workunit.client.0.smithi110.stdout:test delete_pools_parallel on pid 40821 2024-01-27T03:44:49.758 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40821 2024-01-27T03:44:49.758 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.758 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s cls 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' cls' 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.759 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.760 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.761 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40753/exe 2024-01-27T03:44:49.762 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.763 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.764 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.765 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo cls 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.766 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=cls 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.767 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40859 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.768 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.769 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.770 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.770 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test cls on pid 40859' 2024-01-27T03:44:49.770 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.770 INFO:tasks.workunit.client.0.smithi110.stdout:test cls on pid 40859 2024-01-27T03:44:49.781 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40859 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.782 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s cmd 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' cmd' 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo cmd 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40725/exe 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.783 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.784 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.785 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.786 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.787 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.788 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.789 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.790 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.791 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.791 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40772/exe 2024-01-27T03:44:49.791 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=cmd 2024-01-27T03:44:49.791 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40900 2024-01-27T03:44:49.791 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test cmd on pid 40900' 2024-01-27T03:44:49.791 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-01-27T03:44:49.791 INFO:tasks.workunit.client.0.smithi110.stdout:test cmd on pid 40900 2024-01-27T03:44:49.795 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40900 2024-01-27T03:44:49.795 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.795 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s handler_error 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' handler_error' 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo handler_error 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.796 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.797 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=handler_error 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40915 2024-01-27T03:44:49.798 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test handler_error on pid 40915' 2024-01-27T03:44:49.799 INFO:tasks.workunit.client.0.smithi110.stdout:test handler_error on pid 40915 2024-01-27T03:44:49.807 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40915 2024-01-27T03:44:49.807 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.807 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.807 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.807 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.807 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_list_parallel.log 2024-01-27T03:44:49.807 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.807 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ list_parallel: /' 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s io 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' io' 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.808 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo io 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.809 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.810 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.811 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.812 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.813 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=io 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40953 2024-01-27T03:44:49.814 INFO:tasks.workunit.client.0.smithi110.stdout:test io on pid 40953 2024-01-27T03:44:49.825 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test io on pid 40953' 2024-01-27T03:44:49.825 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40953 2024-01-27T03:44:49.825 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.825 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.825 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.826 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.827 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.828 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.829 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s ec_io 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' ec_io' 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40795/exe 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.830 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.831 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.832 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.833 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo ec_io 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.834 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.835 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.835 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40821/exe 2024-01-27T03:44:49.835 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=ec_io 2024-01-27T03:44:49.835 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40992 2024-01-27T03:44:49.835 INFO:tasks.workunit.client.0.smithi110.stdout:test ec_io on pid 40992 2024-01-27T03:44:49.842 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test ec_io on pid 40992' 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=40992 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.843 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.844 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.845 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s list 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' list' 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.846 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo list 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.847 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=list 2024-01-27T03:44:49.848 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41025 2024-01-27T03:44:49.849 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test list on pid 41025' 2024-01-27T03:44:49.849 INFO:tasks.workunit.client.0.smithi110.stdout:test list on pid 41025 2024-01-27T03:44:49.862 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=41025 2024-01-27T03:44:49.862 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.862 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.862 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-01-27T03:44:49.862 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.862 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.862 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.862 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.862 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.863 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.864 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.865 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.866 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.867 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.868 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.869 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.870 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s ec_list 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' ec_list' 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40915/exe 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.871 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.872 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout:01-27T03:44:49.542+0000 7fa0b05fa780 10 monclient: _renew_subs 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0ac8fb640 4 set_mon_vals no callback set 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0b05fa780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55eb8d0dfe90 con 0x55eb8d1be480 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0b05fa780 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0b05fa780 10 monclient: _renew_subs 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0b05fa780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb8d0d3040 con 0x55eb8d1be480 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0b05fa780 1 librados: init done 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.542+0000 7fa0b05fa780 10 librados: wait_for_osdmap waiting 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.632+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fa0a406a030 con 0x55eb8d1be480 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.632+0000 7fa09f7fe640 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa09006abc0 0x7fa09006d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.633+0000 7fa0ad8fd640 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa09006abc0 0x7fa09006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:44:49.873 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.633+0000 7fa0ad8fd640 10 monclient: get_auth_request con 0x7fa09006abc0 auth_method 0 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.633+0000 7fa0ad8fd640 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa09006abc0 0x7fa09006d080 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55eb8d24d2b0 tx=0x7fa0a0001500 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.633+0000 7fa09f7fe640 10 client.4423.objecter ms_handle_connect 0x7fa09006abc0 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.634+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa0a4068070 con 0x55eb8d1be480 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.634+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(26..26 src has 1..26) v4 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.634+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [26,26] > 0 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.634+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding full epoch 26 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.634+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.634+0000 7fa0b05fa780 10 librados: wait_for_osdmap done waiting 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.634+0000 7fa0b05fa780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:44:49.874 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.634+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55eb8d133aa0 con 0x55eb8d1be480 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.878 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.879 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.880 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.881 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.881 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.881 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.881 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.881 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.881 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo ec_list 2024-01-27T03:44:49.881 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.881 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=ec_list 2024-01-27T03:44:49.881 INFO:tasks.workunit.client.0.smithi110.stdout: test ec_list on pid 41064 2024-01-27T03:44:49.884 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41064 2024-01-27T03:44:49.884 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test ec_list on pid 41064' 2024-01-27T03:44:49.884 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=41064 2024-01-27T03:44:49.884 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.884 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.884 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s misc 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' misc' 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40900/exe 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo misc 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.885 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.886 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_cmd 2024-01-27T03:44:49.887 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_cmd.log 2024-01-27T03:44:49.888 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=misc 2024-01-27T03:44:49.888 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41085 2024-01-27T03:44:49.888 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test misc on pid 41085' 2024-01-27T03:44:49.888 INFO:tasks.workunit.client.0.smithi110.stdout:test misc on pid 41085 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=41085 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s pool 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' pool' 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40859/exe 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.907 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.908 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.909 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_cls 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.910 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo pool 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.911 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_handler_error 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.912 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.913 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.914 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40953/exe 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_handler_error.log 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ handler_error: /' 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.915 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.916 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ cmd: /' 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_io.log 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.917 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.918 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.919 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_cls.log 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.920 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ cls: /' 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.921 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_io 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ io: /' 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/41085/exe 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=pool 2024-01-27T03:44:49.922 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41149 2024-01-27T03:44:49.923 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test pool on pid 41149' 2024-01-27T03:44:49.923 INFO:tasks.workunit.client.0.smithi110.stdout:test pool on pid 41149 2024-01-27T03:44:49.932 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-01-27T03:44:49.932 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=41149 2024-01-27T03:44:49.932 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.932 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.932 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s read_operations 2024-01-27T03:44:49.932 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' read_operations' 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.933 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.934 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo read_operations 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.935 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ misc: /' 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.936 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/40992/exe 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.937 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.938 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_misc 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_ec_io 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_misc.log 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ ec_io: /' 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=read_operations 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41187 2024-01-27T03:44:49.939 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test read_operations on pid 41187' 2024-01-27T03:44:49.940 INFO:tasks.workunit.client.0.smithi110.stdout:test read_operations on pid 41187 2024-01-27T03:44:49.949 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=41187 2024-01-27T03:44:49.949 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_ec_io.log 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.950 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s snapshots 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.951 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' snapshots' 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.952 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.953 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.954 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/41025/exe 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo snapshots 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.955 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.956 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.957 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_list 2024-01-27T03:44:49.958 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=snapshots 2024-01-27T03:44:49.958 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/41064/exe 2024-01-27T03:44:49.958 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_list.log 2024-01-27T03:44:49.958 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41222 2024-01-27T03:44:49.958 INFO:tasks.workunit.client.0.smithi110.stdout:test snapshots on pid 41222 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test snapshots on pid 41222' 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=41222 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ list: /' 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.972 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.973 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:49.974 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ ec_list: /' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.975 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:49.976 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.977 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s watch_notify 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_ec_list.log 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_ec_list 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' watch_notify' 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.978 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.979 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.980 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.981 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo watch_notify 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/41149/exe 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:49.982 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_pool.log 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_pool 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=watch_notify 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41269 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:49.983 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test watch_notify on pid 41269' 2024-01-27T03:44:49.984 INFO:tasks.workunit.client.0.smithi110.stdout:test watch_notify on pid 41269 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=41269 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ pool: /' 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:++ printf %25s write_operations 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:+ r=' write_operations' 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:49.987 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:49.988 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:++ echo write_operations 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:++ awk '{print $1}' 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:+ ff=write_operations 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41277 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'test write_operations on pid 41277' 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:49.989 INFO:tasks.workunit.client.0.smithi110.stdout:test write_operations on pid 41277 2024-01-27T03:44:50.014 INFO:tasks.workunit.client.0.smithi110.stderr:+ pids[$f]=41277 2024-01-27T03:44:50.014 INFO:tasks.workunit.client.0.smithi110.stderr:+ ret=0 2024-01-27T03:44:50.014 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' 1 -eq 1 ']' 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40284 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40284 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/41187/exe 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -f /etc/bashrc ']' 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.015 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:+ . /etc/bashrc 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -z '' ']' 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++ BASHRCSOURCED=Y 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++ shopt -q login_shell 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:50.016 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.017 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_read_operations.log 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ read_operations: /' 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_read_operations 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:+++ umask 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' 0022 -eq 0 ']' 2024-01-27T03:44:50.018 INFO:tasks.workunit.client.0.smithi110.stderr:++ SHELL=/bin/bash 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:50.019 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.020 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:50.021 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:50.022 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'grep=grep --color=auto' 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorls.sh 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' '!' -t 0 ']' 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:+++ return 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export S_COLORS=auto 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:+++ S_COLORS=auto 2024-01-27T03:44:50.023 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:50.024 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.025 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/41222/exe 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-27T03:44:50.026 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.027 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_snapshots 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ snapshots: /' 2024-01-27T03:44:50.028 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_snapshots.log 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-27T03:44:50.029 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/gawk.sh 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/lang.sh 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:50.030 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/locale 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.031 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/41269/exe 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:++++ export LANG=en_US.UTF-8 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:++++ LANG=en_US.UTF-8 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:50.032 INFO:tasks.workunit.client.0.smithi110.stderr:+++ LANG=en_US.UTF-8 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:+++ unset LANG_backup config 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n '' ']' 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' dumb = linux ']' 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/less.sh 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:++ for i in /etc/profile.d/*.sh 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:++ '[' '' ']' 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:++ . /etc/profile.d/which2.sh 2024-01-27T03:44:50.033 INFO:tasks.workunit.client.0.smithi110.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+++++ readlink /proc/41277/exe 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_watch_notify 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ watch_notify: /' 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:++++ basename /usr/bin/bash 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_declare='declare -f' 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+++ which_opt=-f 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export which_declare 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:+++ export -f which 2024-01-27T03:44:50.034 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset i 2024-01-27T03:44:50.035 INFO:tasks.workunit.client.0.smithi110.stderr:++ unset -f pathmunge 2024-01-27T03:44:50.035 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-27T03:44:50.035 INFO:tasks.workunit.client.0.smithi110.stderr:+ export PATH 2024-01-27T03:44:50.035 INFO:tasks.workunit.client.0.smithi110.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-27T03:44:50.035 INFO:tasks.workunit.client.0.smithi110.stderr:+ unset rc 2024-01-27T03:44:50.035 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph_test_neorados_write_operations 2024-01-27T03:44:50.035 INFO:tasks.workunit.client.0.smithi110.stderr:+ tee ceph_test_neorados_write_operations.log 2024-01-27T03:44:50.035 INFO:tasks.workunit.client.0.smithi110.stderr:+ sed 's/^/ write_operations: /' 2024-01-27T03:44:52.533 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.691+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0a4095ac0 con 0x55eb8d1be480 2024-01-27T03:44:52.533 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.691+0000 7fa09f7fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-27T03:44:52.533 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.691+0000 7fa09ffff640 10 client.4423.objecter _get_latest_version latest 26, have it 2024-01-27T03:44:52.533 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.691+0000 7fa0b05fa780 10 client.4423.objecter create_pool name=ceph_test_rados_api_asio 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.691+0000 7fa0b05fa780 10 client.4423.objecter pool_op_submit 1 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.691+0000 7fa0b05fa780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:49.691+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55eb8d144ca0 con 0x55eb8d1be480 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.493+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.493+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.493+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.493+0000 7fa09e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== mon.1 v2:172.21.15.162:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v28) v1 ==== 43+0+0 (secure 0 0 0) 0x7fa0a4057050 con 0x55eb8d1be480 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 pool_op_reply(tid 1 (0) Success v28) v1 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 10 client.4423.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v28) v1 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 10 client.4423.objecter have request 1 at 0x55eb8d24be30 Op: create 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 20 client.4423.objecter waiting for client to reach epoch 28 before calling back 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 10 monclient: _renew_subs 2024-01-27T03:44:52.534 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:44:52.535 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=27}) v3 -- 0x7fa0900708d0 con 0x55eb8d1be480 2024-01-27T03:44:52.535 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.496+0000 7fa09f7fe640 10 client.4423.objecter done 2024-01-27T03:44:52.535 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== mon.1 v2:172.21.15.162:3300/0 8 ==== osd_map(27..28 src has 1..28) v4 ==== 15340+0+0 (secure 0 0 0) 0x7fa0a4099f20 con 0x55eb8d1be480 2024-01-27T03:44:52.535 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(27..28 src has 1..28) v4 2024-01-27T03:44:52.535 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [27,28] > 26 2024-01-27T03:44:52.535 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 27 2024-01-27T03:44:52.535 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 28 2024-01-27T03:44:52.535 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 10 librados: set snap read head -> 1 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 10 librados: writefull oid=exist nspace= 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d24e0b0 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 28 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x55eb8d24ad10 0x55eb8d24b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 2 osd.7 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:44:55.495 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 2 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 15 client.4423.objecter _send_op 2 to 9.1 on osd.7 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:2 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) v8 -- 0x55eb8d24b630 con 0x55eb8d24ad10 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.532+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.585+0000 7fa0ae8ff640 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x55eb8d24ad10 0x55eb8d24b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.592+0000 7fa0ae8ff640 10 monclient: get_auth_request con 0x55eb8d24ad10 auth_method 0 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.658+0000 7fa09f7fe640 10 client.4423.objecter ms_handle_connect 0x55eb8d24ad10 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:52.658+0000 7fa0ae8ff640 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x55eb8d24ad10 0x55eb8d24b0f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:53.493+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:53.493+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:53.493+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:54.493+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:44:55.496 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:54.493+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:44:55.497 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:54.493+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:44:55.497 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:54.543+0000 7fa0b0362640 10 client.4423.objecter tick 2024-01-27T03:44:55.497 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:55.494+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:01.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:55.494+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:01.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:55.494+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:01.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:56.497+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:01.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:56.497+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:56.497+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:57.497+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:57.497+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:57.497+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:58.497+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:58.497+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:58.497+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:59.497+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:59.497+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:59.497+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:44:59.544+0000 7fa0b0362640 10 client.4423.objecter tick 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:00.497+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:00.497+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:00.497+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:01.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 1 ==== osd_map(29..31 src has 1..31) v4 ==== 9922+0+0 (crc 0 0 0) 0x7fa0a80026b0 con 0x55eb8d24ad10 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(29..31 src has 1..31) v4 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [29,31] > 28 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 29 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 10 client.4423.objecter checking op 2 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 29 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 30 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 10 client.4423.objecter checking op 2 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 30 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 31 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 10 client.4423.objecter checking op 2 2024-01-27T03:45:01.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 31 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:04.552 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:04.552 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:45:04.552 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.388+0000 7fa09f7fe640 20 client.4423.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-27T03:45:04.552 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.497+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:04.552 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.497+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:04.552 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:01.497+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:02.497+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:02.497+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:02.497+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:02.497+0000 7fa09e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:03.497+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:03.497+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:03.497+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.545+0000 7fa0b0362640 10 client.4423.objecter tick 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.545+0000 7fa0b0362640 2 client.4423.objecter tid 2 on osd.7 is laggy 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.545+0000 7fa0b0362640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T03:45:04.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.545+0000 7fa0b0362640 10 monclient: _renew_subs 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.545+0000 7fa0b0362640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.545+0000 7fa0b0362640 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=32}) v3 -- 0x7fa080002f90 con 0x55eb8d1be480 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.545+0000 7fa0b0362640 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- ping magic: 0 v1 -- 0x7fa080003320 con 0x55eb8d24ad10 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== mon.1 v2:172.21.15.162:3300/0 9 ==== osd_map(32..36 src has 1..36) v4 ==== 9879+0+0 (secure 0 0 0) 0x7fa0a40997b0 con 0x55eb8d1be480 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(32..36 src has 1..36) v4 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [32,36] > 31 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 32 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 10 client.4423.objecter checking op 2 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 32 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 33 2024-01-27T03:45:04.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 10 client.4423.objecter checking op 2 2024-01-27T03:45:04.730 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 33 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:04.730 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:04.730 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 34 2024-01-27T03:45:04.730 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 10 client.4423.objecter checking op 2 2024-01-27T03:45:04.730 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 34 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:04.730 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 35 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 10 client.4423.objecter checking op 2 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 35 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 36 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 10 client.4423.objecter checking op 2 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.551+0000 7fa09f7fe640 20 client.4423.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 2 ==== osd_op_reply(2 exist [writefull 0~5] v31'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(2 exist [writefull 0~5] v31'1 uv1 ondisk = 0) v8 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 2 ondisk uv 1 in 9.1 attempt 0 2024-01-27T03:45:04.731 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:04.732 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 2 2024-01-27T03:45:04.732 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 2 2024-01-27T03:45:04.732 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:04.732 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 2 2024-01-27T03:45:04.732 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:04.732 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-01-27T03:45:04.732 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 10 librados: Objecter returned from writefull r=0 2024-01-27T03:45:04.732 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d24e0b0 2024-01-27T03:45:04.732 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:07.499 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:07.499 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:07.499 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:07.499 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 3 osd.7 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 3 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 15 client.4423.objecter _send_op 3 to 9.1 on osd.7 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:3 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55eb8d2ebe20 con 0x55eb8d24ad10 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d2ec1a0 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 36 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 4 osd.7 2024-01-27T03:45:07.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 4 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 15 client.4423.objecter _send_op 4 to 9.1 on osd.7 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:4 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55eb8d2ec640 con 0x55eb8d24ad10 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:04.729+0000 7fa0b05fa780 5 client.4423.objecter 2 in flight 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:05.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:05.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:05.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:06.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:06.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:06.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:07.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:07.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:10.028 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:07.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:10.028 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:07.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:10.028 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:08.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:10.028 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:08.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:10.028 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:08.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:09.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:09.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:09.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:09.546+0000 7fa0b0362640 10 client.4423.objecter tick 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 3 ondisk uv 1 in 9.1 attempt 0 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 5 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 3 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 3 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 3 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 5 client.4423.objecter 1 in flight 2024-01-27T03:45:10.029 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (5298 ms) 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 4 ondisk uv 0 in 9.1 attempt 0 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 4 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 4 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 4 2024-01-27T03:45:10.030 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:10.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d2ec1a0 2024-01-27T03:45:10.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:10.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:10.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:10.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:10.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 5 osd.7 2024-01-27T03:45:10.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:10.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 5 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 15 client.4423.objecter _send_op 5 to 9.1 on osd.7 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:5 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55eb8d2ec9d0 con 0x55eb8d24ad10 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d24e0b0 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 36 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 6 osd.7 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 6 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 15 client.4423.objecter _send_op 6 to 9.1 on osd.7 2024-01-27T03:45:10.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:6 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55eb8d2fb870 con 0x55eb8d24ad10 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.027+0000 7fa0b05fa780 5 client.4423.objecter 2 in flight 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (39 ms) 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 5 ondisk uv 1 in 9.1 attempt 0 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 5 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 5 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 5 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 5 2024-01-27T03:45:10.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 5 client.4423.objecter 1 in flight 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 6 ondisk uv 0 in 9.1 attempt 0 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 6 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 6 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 6 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.065+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d2ec180 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:10.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:10.071 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:10.071 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:10.071 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 7 osd.7 2024-01-27T03:45:10.071 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:10.071 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 7 2024-01-27T03:45:10.071 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 15 client.4423.objecter _send_op 7 to 9.1 on osd.7 2024-01-27T03:45:10.345 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:7 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55eb8d24e1e0 con 0x55eb8d24ad10 2024-01-27T03:45:10.345 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:10.345 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:10.345 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d22c970 2024-01-27T03:45:10.345 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 36 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:10.345 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-01-27T03:45:10.345 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 8 osd.7 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 8 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 15 client.4423.objecter _send_op 8 to 9.1 on osd.7 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:8 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) v8 -- 0x55eb8d22ce10 con 0x55eb8d24ad10 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.066+0000 7fa0b05fa780 5 client.4423.objecter 2 in flight 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 7 ondisk uv 1 in 9.1 attempt 0 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 5 2024-01-27T03:45:10.346 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 7 2024-01-27T03:45:10.347 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 7 2024-01-27T03:45:10.347 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:10.347 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 7 2024-01-27T03:45:10.347 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 5 client.4423.objecter 1 in flight 2024-01-27T03:45:10.347 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:11.499 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-27T03:45:11.499 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 8 ondisk uv 0 in 9.1 attempt 0 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 8 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 8 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 8 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (278 ms) 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 20 librados: queue_aio_write 0x55eb8d0fe890 completion 0x55eb8d0ce3c0 write_seq 1 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d22c970 2024-01-27T03:45:11.500 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 9 osd.7 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 9 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 15 client.4423.objecter _send_op 9 to 9.1 on osd.7 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:9 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36) v8 -- 0x55eb8d2ec180 con 0x55eb8d24ad10 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.344+0000 7fa0b05fa780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:11.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:10.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:11.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:11.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:12.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:12.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 9 ==== osd_map(37..38 src has 1..38) v4 ==== 1244+0+0 (crc 0 0 0) 0x7fa0a8002bc0 con 0x55eb8d24ad10 2024-01-27T03:45:12.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(37..38 src has 1..38) v4 2024-01-27T03:45:12.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [37,38] > 36 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 37 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 10 client.4423.objecter checking op 9 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 37 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 38 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 10 client.4423.objecter checking op 9 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:11.689+0000 7fa09f7fe640 20 client.4423.objecter 9 9.d3a59201 osd.7 exist [write 0~5 in=5b] 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 10 ==== osd_op_reply(9 exist [write 0~5] v38'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(9 exist [write 0~5] v38'2 uv2 ondisk = 0) v8 2024-01-27T03:45:12.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 9 ondisk uv 2 in 9.1 attempt 0 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 9 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 9 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 9 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0ae8ff640 20 librados: complete_aio_write 0x55eb8d0ce3c0 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (1949 ms) 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 20 librados: queue_aio_write 0x55eb8d0fe890 completion 0x55eb8d0dad80 write_seq 2 2024-01-27T03:45:12.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d22c950 2024-01-27T03:45:12.622 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:12.622 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:12.622 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:12.622 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:12.622 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 10 osd.7 2024-01-27T03:45:12.622 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:12.622 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 10 2024-01-27T03:45:12.622 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 15 client.4423.objecter _send_op 10 to 9.1 on osd.7 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:10 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38) v8 -- 0x55eb8d22c070 con 0x55eb8d24ad10 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.293+0000 7fa0b05fa780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.498+0000 7fa09e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 11 ==== osd_map(39..40 src has 1..40) v4 ==== 2038+0+0 (crc 0 0 0) 0x7fa0a8002d80 con 0x55eb8d24ad10 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(39..40 src has 1..40) v4 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [39,40] > 38 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 39 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 10 client.4423.objecter checking op 10 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:12.623 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:12.624 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 40 2024-01-27T03:45:12.624 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 10 client.4423.objecter checking op 10 2024-01-27T03:45:12.624 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 40 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:12.624 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:12.624 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:45:14.442 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:00.398 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:14.446 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:00.402 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:14.514 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:03:00.460 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:14.518 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:03:00.464 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:14.524 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:00.480 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:14.529 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:00.486 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:14.796 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:00.752 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:14.804 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:03:00.750 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:14.917 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:00.874 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:12.621+0000 7fa09f7fe640 20 client.4423.objecter 10 9.d3a59201 osd.7 exist [write 0~5 in=5b] 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:13.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:13.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:13.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:14.498+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:14.498+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:14.498+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:14.547+0000 7fa0b0362640 10 client.4423.objecter tick 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.499+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:15.807 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.499+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.499+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 12 ==== osd_op_reply(10 exist [write 0~5] v40'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(10 exist [write 0~5] v40'3 uv3 ondisk = 0) v8 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 10 ondisk uv 3 in 9.1 attempt 0 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 10 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 10 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 10 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.805+0000 7fa0ae8ff640 20 librados: complete_aio_write 0x55eb8d0dad80 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3513 ms) 2024-01-27T03:45:15.808 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-01-27T03:45:15.809 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-01-27T03:45:15.809 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 20 librados: queue_aio_write 0x55eb8d0fe890 completion 0x55eb8d22c560 write_seq 3 2024-01-27T03:45:15.809 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d22c950 2024-01-27T03:45:15.809 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 40 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:15.809 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:15.809 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:15.809 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:15.809 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 11 osd.7 2024-01-27T03:45:15.809 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15. pool: Running main() from gmock_main.cc 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [----------] Global test environment set-up. 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ OK ] NeoRadosPools.PoolList (6461 ms) 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3926 ms) 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2696 ms) 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (7200 ms) 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2484 ms) 2024-01-27T03:45:15.877 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-01-27T03:45:15.878 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (3302 ms) 2024-01-27T03:45:15.878 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [----------] 6 tests from NeoRadosPools (26069 ms total) 2024-01-27T03:45:15.878 INFO:tasks.workunit.client.0.smithi110.stdout: pool: 2024-01-27T03:45:15.878 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [----------] Global test environment tear-down 2024-01-27T03:45:15.878 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [==========] 6 tests from 1 test suite ran. (26069 ms total) 2024-01-27T03:45:15.878 INFO:tasks.workunit.client.0.smithi110.stdout: pool: [ PASSED ] 6 tests. 2024-01-27T03:45:17.861 INFO:tasks.workunit.client.0.smithi110.stdout:806+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 11 2024-01-27T03:45:17.861 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 15 client.4423.objecter _send_op 11 to 9.1 on osd.7 2024-01-27T03:45:17.861 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:11 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v8 -- 0x55eb8d230770 con 0x55eb8d24ad10 2024-01-27T03:45:17.861 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.861 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:17.861 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:15.806+0000 7fa0b05fa780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 13 ==== osd_map(41..43 src has 1..43) v4 ==== 2632+0+0 (crc 0 0 0) 0x7fa0a8021360 con 0x55eb8d24ad10 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(41..43 src has 1..43) v4 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [41,43] > 40 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 41 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 10 client.4423.objecter checking op 11 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 41 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 42 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 10 client.4423.objecter checking op 11 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 42 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 43 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 10 client.4423.objecter checking op 11 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 43 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:17.862 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.121+0000 7fa09f7fe640 20 client.4423.objecter 11 9.d3a59201 osd.7 exist [write 0~5 in=5b] 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.499+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.499+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:16.499+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.499+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.499+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.499+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 14 ==== osd_op_reply(11 exist [write 0~5] v43'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(11 exist [write 0~5] v43'4 uv4 ondisk = 0) v8 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 11 ondisk uv 4 in 9.1 attempt 0 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:17.863 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (2054 ms) 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 11 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 11 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 11 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.859+0000 7fa0ae8ff640 20 librados: complete_aio_write 0x55eb8d22c560 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d22c950 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 43 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 12 osd.7 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:17.864 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 12 2024-01-27T03:45:17.865 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 15 client.4423.objecter _send_op 12 to 9.1 on osd.7 2024-01-27T03:45:17.865 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:12 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55eb8d230100 con 0x55eb8d24ad10 2024-01-27T03:45:17.865 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.865 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:17.865 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d230460 2024-01-27T03:45:17.865 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 43 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:17.865 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-01-27T03:45:17.865 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:17.865 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.929 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 13 osd.7 2024-01-27T03:45:17.929 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.929 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 13 2024-01-27T03:45:17.929 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 15 client.4423.objecter _send_op 13 to 9.1 on osd.7 2024-01-27T03:45:17.929 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:13 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43) v8 -- 0x55eb8d22d170 con 0x55eb8d24ad10 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.860+0000 7fa0b05fa780 5 client.4423.objecter 2 in flight 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.927+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 15 ==== osd_map(44..44 src has 1..44) v4 ==== 1031+0+0 (crc 0 0 0) 0x7fa0a8021a30 con 0x55eb8d24ad10 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.927+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 16 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.927+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(44..44 src has 1..44) v4 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [44,44] > 43 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 44 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 10 client.4423.objecter checking op 12 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 44 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 10 client.4423.objecter checking op 13 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 44 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:17.930 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 20 client.4423.objecter 12 9.d3a59201 osd.7 exist [read 0~0] 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa09f7fe640 20 client.4423.objecter 13 9.a7bbaa1 osd.7 noexist [read 0~0] 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 12 ondisk uv 4 in 9.1 attempt 0 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 5 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 12 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 12 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 12 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 5 client.4423.objecter 1 in flight 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 17 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:17.931 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 13 ondisk uv 0 in 9.1 attempt 0 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 13 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 13 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 13 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (68 ms) 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d230100 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 44 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:17.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:17.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 14 osd.7 2024-01-27T03:45:17.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:17.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 14 2024-01-27T03:45:17.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 15 client.4423.objecter _send_op 14 to 9.1 on osd.7 2024-01-27T03:45:17.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:14 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) v8 -- 0x55eb8d22ca60 con 0x55eb8d24ad10 2024-01-27T03:45:17.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:17.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d22d170 2024-01-27T03:45:17.967 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 44 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:17.967 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 15 osd.7 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 15 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 15 client.4423.objecter _send_op 15 to 9.1 on osd.7 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:15 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) v8 -- 0x55eb8d230740 con 0x55eb8d24ad10 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.928+0000 7fa0b05fa780 5 client.4423.objecter 2 in flight 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (38 ms) 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 18 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:17.968 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 14 ondisk uv 4 in 9.1 attempt 0 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 5 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 14 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 14 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 14 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 5 client.4423.objecter 1 in flight 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 19 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 15 ondisk uv 0 in 9.1 attempt 0 2024-01-27T03:45:17.969 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 15 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 15 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 15 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d230100 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 44 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 16 osd.7 2024-01-27T03:45:17.970 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 16 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 15 client.4423.objecter _send_op 16 to 9.1 on osd.7 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:16 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) v8 -- 0x55eb8d22ca60 con 0x55eb8d24ad10 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d22d930 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 44 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 17 osd.7 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 17 2024-01-27T03:45:17.971 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 15 client.4423.objecter _send_op 17 to 9.1 on osd.7 2024-01-27T03:45:18.087 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:17 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) v8 -- 0x55eb8d22ddd0 con 0x55eb8d24ad10 2024-01-27T03:45:18.087 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:18.087 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:17.966+0000 7fa0b05fa780 5 client.4423.objecter 2 in flight 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 20 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 16 ondisk uv 4 in 9.1 attempt 0 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 5 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 16 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 16 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 5 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 16 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 5 client.4423.objecter 1 in flight 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 21 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 17 ondisk uv 0 in 9.1 attempt 0 2024-01-27T03:45:18.088 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 10 client.4423.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 17 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 17 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 17 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (120 ms) 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 20 librados: queue_aio_write 0x55eb8d0fe890 completion 0x55eb8d230880 write_seq 4 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d230100 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 44 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:18.089 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_asi api_service: [==========] Running 4 tests from 1 test suite. 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [----------] Global test environment set-up. 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [----------] 4 tests from LibRadosService 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5467 ms) 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [ OK ] LibRadosService.RegisterLate (62 ms) 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: cluster: 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: id: 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: health: HEALTH_WARN 2024-01-27T03:45:19.207 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 32 pool(s) do not have an application enabled 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: services: 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: osd: 8 osds: 8 up (since 95s), 8 in (since 2m) 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: data: 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: pools: 37 pools, 1129 pgs 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: objects: 2 objects, 449 KiB 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: pgs: 94.331% pgs unknown 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 4.872% pgs not active 2024-01-27T03:45:19.208 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 1065 unknown 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 41 creating+peering 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 14 creating+activating 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 9 active+clean 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: cluster: 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: id: 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: health: HEALTH_WARN 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 17 pool(s) do not have an application enabled 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: services: 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: osd: 8 osds: 8 up (since 97s), 8 in (since 2m) 2024-01-27T03:45:19.209 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: data: 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: pools: 36 pools, 1025 pgs 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: objects: 2 objects, 449 KiB 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: pgs: 69.854% pgs unknown 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 28.780% pgs not active 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 716 unknown 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 266 creating+peering 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 29 creating+activating 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 14 active+clean 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 2024-01-27T03:45:19.210 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 2024-01-27T03:45:19.211 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [ OK ] LibRadosService.StatusFormat (3958 ms) 2024-01-27T03:45:19.211 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [ RUN ] LibRadosService.Status 2024-01-27T03:45:19.211 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [ OK ] LibRadosService.Status (20123 ms) 2024-01-27T03:45:19.211 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [----------] 4 tests from LibRadosService (29610 ms total) 2024-01-27T03:45:19.211 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: 2024-01-27T03:45:19.211 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [----------] Global test environment tear-down 2024-01-27T03:45:19.211 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29610 ms total) 2024-01-27T03:45:19.211 INFO:tasks.workunit.client.0.smithi110.stdout: api_service: [ PASSED ] 4 tests. 2024-01-27T03:45:20.802 INFO:tasks.workunit.client.0.smithi110.stdout:o: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:20.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:20.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 18 osd.7 2024-01-27T03:45:20.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:20.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 18 2024-01-27T03:45:20.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 15 client.4423.objecter _send_op 18 to 9.1 on osd.7 2024-01-27T03:45:20.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:18 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) v8 -- 0x55eb8d22ca60 con 0x55eb8d24ad10 2024-01-27T03:45:20.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:20.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.086+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.499+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.499+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:18.499+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:19.499+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:19.499+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:19.499+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:19.548+0000 7fa0b0362640 10 client.4423.objecter tick 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.499+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.499+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.499+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.800+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 22 ==== osd_op_reply(18 exist [writefull 0~5] v44'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.800+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(18 exist [writefull 0~5] v44'5 uv5 ondisk = 0) v8 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.800+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.800+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 18 ondisk uv 5 in 9.1 attempt 0 2024-01-27T03:45:20.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.800+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:20.804 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.800+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 18 2024-01-27T03:45:20.804 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.800+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 18 2024-01-27T03:45:20.804 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:20.804 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 18 2024-01-27T03:45:20.804 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:20.804 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0ae8ff640 20 librados: complete_aio_write 0x55eb8d230880 2024-01-27T03:45:20.804 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2715 ms) 2024-01-27T03:45:21.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-01-27T03:45:21.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [----------] Global test environment set-up. 2024-01-27T03:45:21.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5408 ms) 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (486 ms) 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20258 ms) 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: attempt 0 of 20 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: failed to deregister: 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: { 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "epoch": 12, 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "modified": "2024-01-27T03:45:16.876426+0000", 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "services": { 2024-01-27T03:45:21.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "laundry": { 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "daemons": { 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "summary": "", 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "close-test-pid40767": { 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "start_epoch": 12, 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "start_stamp": "2024-01-27T03:45:15.932269+0000", 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "gid": 4666, 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "addr": "172.21.15.110:0/2599607665", 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "metadata": { 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "arch": "x86_64", 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "ceph_release": "squid", 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-926-g5ffe0641 (5ffe0641f18e6ec692535294010d9456630e7923) squid (dev)", 2024-01-27T03:45:21.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "ceph_version_short": "19.0.0-926-g5ffe0641", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "distro": "centos", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "distro_version": "9", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "foo": "bar", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "hostname": "smithi110", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Mon Jan 22 21:08:16 UTC 2024", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "kernel_version": "5.14.0-411.el9.x86_64", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "mem_swap_kb": "0", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "mem_total_kb": "32488368", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "os": "Linux", 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "this": "that" 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: }, 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "task_status": {} 2024-01-27T03:45:21.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: }, 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "pid40712": { 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "start_epoch": 11, 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "start_stamp": "2024-01-27T03:44:59.212089+0000", 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "gid": 4580, 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "addr": "172.21.15.110:0/313782187", 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "metadata": { 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "arch": "x86_64", 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "ceph_release": "squid", 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-926-g5ffe0641 (5ffe0641f18e6ec692535294010d9456630e7923) squid (dev)", 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "ceph_version_short": "19.0.0-926-g5ffe0641", 2024-01-27T03:45:21.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 20 librados: queue_aio_write 0x55eb8d0fe890 completion 0x55eb8d22c950 write_seq 5 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d230100 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 44 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 19 osd.7 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 19 2024-01-27T03:45:21.501 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 15 client.4423.objecter _send_op 19 to 9.1 on osd.7 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:19 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) v8 -- 0x55eb8d22ca60 con 0x55eb8d24ad10 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.801+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 23 ==== osd_map(45..46 src has 1..46) v4 ==== 1667+0+0 (crc 0 0 0) 0x7fa0a8021d10 con 0x55eb8d24ad10 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(45..46 src has 1..46) v4 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [45,46] > 44 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 45 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 10 client.4423.objecter checking op 19 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 45 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 46 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 10 client.4423.objecter checking op 19 2024-01-27T03:45:21.502 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 46 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:21.503 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:21.503 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:45:21.503 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:20.815+0000 7fa09f7fe640 20 client.4423.objecter 19 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-27T03:45:21.503 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:21.499+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:22.697 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:21.499+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:22.697 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:21.499+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:22.697 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.499+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:22.697 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.499+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:22.697 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.499+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:22.697 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.500+0000 7fa09e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T03:45:22.697 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 24 ==== osd_op_reply(19 exist [writefull 0~5] v46'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:22.697 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(19 exist [writefull 0~5] v46'6 uv6 ondisk = 0) v8 2024-01-27T03:45:22.697 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 19 ondisk uv 6 in 9.1 attempt 0 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 19 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 19 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 19 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0ae8ff640 20 librados: complete_aio_write 0x55eb8d22c950 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1895 ms) 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 20 librados: queue_aio_write 0x55eb8d0fe890 completion 0x55eb8d22dfd0 write_seq 6 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 10 client.4423.objecter _op_submit op 0x55eb8d230100 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 20 client.4423.objecter _calc_target epoch 46 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:22.698 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 10 client.4423.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-01-27T03:45:22.699 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 20 client.4423.objecter _get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:22.699 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 10 client.4423.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 20 osd.7 2024-01-27T03:45:22.699 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 20 client.4423.objecter get_session s=0x55eb8d0f3d80 osd=7 3 2024-01-27T03:45:22.699 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 15 client.4423.objecter _session_op_assign 7 20 2024-01-27T03:45:22.699 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 15 client.4423.objecter _send_op 20 to 9.1 on osd.7 2024-01-27T03:45:22.699 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- osd_op(unknown.0.0:20 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) v8 -- 0x55eb8d22ca60 con 0x55eb8d24ad10 2024-01-27T03:45:23.983 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:23.983 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.695+0000 7fa0b05fa780 5 client.4423.objecter 1 in flight 2024-01-27T03:45:23.983 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 25 ==== osd_map(47..48 src has 1..49) v4 ==== 1984+0+0 (crc 0 0 0) 0x7fa0a8021e40 con 0x55eb8d24ad10 2024-01-27T03:45:23.983 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_map(47..48 src has 1..49) v4 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map got epochs [47,48] > 46 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 47 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 10 client.4423.objecter checking op 20 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 47 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 3 client.4423.objecter handle_osd_map decoding incremental epoch 48 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 10 client.4423.objecter checking op 20 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 20 client.4423.objecter _calc_target epoch 48 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 20 client.4423.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 20 client.4423.objecter dump_active .. 0 homeless 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:22.776+0000 7fa09f7fe640 20 client.4423.objecter 20 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.500+0000 7fa09e7fc640 10 monclient: tick 2024-01-27T03:45:23.984 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.500+0000 7fa09e7fc640 10 monclient: _check_auth_tickets 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.500+0000 7fa09e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 <== osd.7 v2:172.21.15.162:6818/68785612 26 ==== osd_op_reply(20 exist [writefull 0~5] v48'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fa0a8016040 con 0x55eb8d24ad10 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 10 client.4423.objecter ms_dispatch 0x55eb8d12ec10 osd_op_reply(20 exist [writefull 0~5] v48'7 uv7 ondisk = 0) v8 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 10 client.4423.objecter in handle_osd_op_reply 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 7 client.4423.objecter handle_osd_op_reply 20 ondisk uv 7 in 9.1 attempt 0 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 10 client.4423.objecter op 0 rval 0 len 0 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 15 client.4423.objecter handle_osd_op_reply completed tid 20 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 15 client.4423.objecter _finish_op 20 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 4 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 15 client.4423.objecter _session_op_remove 7 20 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 5 client.4423.objecter 0 in flight 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 20 librados: complete_aio_write 0x55eb8d22dfd0 2024-01-27T03:45:23.985 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1286 ms) 2024-01-27T03:45:23.987 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 10 librados: watch_flush enter 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 10 librados: watch_flush exit 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 10 client.4423.objecter close_session for osd.7 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x55eb8d24ad10 msgr2=0x55eb8d24b0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x55eb8d24ad10 0x55eb8d24b0f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 20 client.4423.objecter put_session s=0x55eb8d0f3d80 osd=7 1 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 20 client.4423.objecter shutdown clearing up homeless session... 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 10 client.4423.objecter successfully canceled tick 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa09006abc0 msgr2=0x7fa09006d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fa09006abc0 0x7fa09006d080 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55eb8d24d2b0 tx=0x7fa0a0001500 comp rx=0 tx=0).stop 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 10 monclient: shutdown 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1be480 msgr2=0x55eb8d258c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T03:45:23.988 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 --2- 172.21.15.110:0/882326301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eb8d1be480 0x55eb8d258c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4000c00 tx=0x7fa0a4002fb0 comp rx=0 tx=0).stop 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0ae8ff640 1 -- 172.21.15.110:0/882326301 reap_dead start 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 shutdown_connections 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 >> 172.21.15.110:0/882326301 conn(0x55eb8d1c4ac0 msgr2=0x55eb8d1bb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 shutdown_connections 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 -- 172.21.15.110:0/882326301 wait complete. 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.982+0000 7fa0b05fa780 1 librados: shutdown 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [----------] 12 tests from AsioRados (19253 ms total) 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [----------] Global test environment tear-down 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (34554 ms total) 2024-01-27T03:45:23.989 INFO:tasks.workunit.client.0.smithi110.stdout: api_asio: [ PASSED ] 12 tests. 2024-01-27T03:45:25.056 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:24.997+0000 12a58640 -1 osd.4 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:26.199 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:26.105+0000 12a58640 -1 osd.4 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:26.593 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "distro": "centos", 2024-01-27T03:45:26.593 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-01-27T03:45:26.593 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "distro_version": "9", 2024-01-27T03:45:26.593 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "foo": "bar", 2024-01-27T03:45:26.593 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "hostname": "smithi110", 2024-01-27T03:45:26.593 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Mon Jan 22 21:08:16 UTC 2024", 2024-01-27T03:45:26.593 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "kernel_version": "5.14.0-411.el9.x86_64", 2024-01-27T03:45:26.593 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "mem_swap_kb": "0", 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "mem_total_kb": "32488368", 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "os": "Linux", 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "this": "that" 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: }, 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: "task_status": {} 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: } 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: } 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: } 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: } 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: } 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: attempt 1 of 20 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10823 ms) 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (36975 ms total) 2024-01-27T03:45:26.594 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: 2024-01-27T03:45:26.595 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [----------] Global test environment tear-down 2024-01-27T03:45:26.595 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (36975 ms total) 2024-01-27T03:45:26.595 INFO:tasks.workunit.client.0.smithi110.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-01-27T03:45:27.313 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:27.252+0000 12a58640 -1 osd.4 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:27.544 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:27.515+0000 12a58640 -1 osd.6 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:28.544 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:28.534+0000 12a58640 -1 osd.6 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:28.598 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:28.580+0000 12a58640 -1 osd.4 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:28.699 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:28.678+0000 12a58640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:2 12.15 12:aecc2179:test-rados-api-smithi110-40874-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:29.637 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:29.624+0000 12a58640 -1 osd.4 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:29.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:29.813+0000 12a58640 -1 osd.1 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:2 12.15 12:aecc2179:test-rados-api-smithi110-40874-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:30.170 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:30.161+0000 12a58640 -1 osd.6 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:30.719 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:30.710+0000 12a58640 -1 osd.4 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:30.945 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: Running main() from gmock_main.cc 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: [----------] Global test environment set-up. 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (41214 ms) 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: [----------] 1 test from neocls_handler_error (41214 ms total) 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: [----------] Global test environment tear-down 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: [==========] 1 test from 1 test suite ran. (41214 ms total) 2024-01-27T03:45:30.946 INFO:tasks.workunit.client.0.smithi110.stdout: handler_error: [ PASSED ] 1 test. 2024-01-27T03:45:31.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:31.008+0000 12a58640 -1 osd.1 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:2 12.15 12:aecc2179:test-rados-api-smithi110-40874-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:31.492 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:31.484+0000 12a58640 -1 osd.6 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:31.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-27T03:45:31.370+0000 ea50640 -1 osd.3 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4427.0:2 25.8 25:1618ccd1:test-rados-api-smithi110-40510-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:32.128 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:32.017+0000 12a58640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:32.521 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:32.485+0000 12a58640 -1 osd.6 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:33.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:32.922+0000 12a58640 -1 osd.1 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4494.0:2 12.15 12:aecc2179:test-rados-api-smithi110-40874-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:33.564 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:33.465+0000 12a58640 -1 osd.6 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:33.636 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:33.502+0000 12a58640 -1 osd.4 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:34.528 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:34.516+0000 12a58640 -1 osd.4 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:35.216 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:35.179+0000 12a58640 -1 osd.6 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:35.467 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:21.423 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:35.548 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:21.504 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:35.586 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:21.542 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:35.592 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:21.548 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:35.714 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:35.690+0000 12a58640 -1 osd.4 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:35.763 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:21.717 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:35.863 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:03:21.808 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:35.865 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:03:21.811 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:36.093 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:22.049 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:36.133 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:03:22.079 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:36.556 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:36.458+0000 12a58640 -1 osd.6 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:36.926 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:36.818+0000 12a58640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:37.639 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:37.570+0000 12a58640 -1 osd.6 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:38.105 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:45:38.043+0000 12a58640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:2 33.1a 33:5c6776db:::0.obj:head [write 0~35 in=35b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-27T03:45:38.626 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:38.611+0000 12a58640 -1 osd.6 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:40.073 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:40.009+0000 12a58640 -1 osd.6 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:41.108 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:40.996+0000 12a58640 -1 osd.6 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:41.984 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:41.976+0000 12a58640 -1 osd.6 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:43.366 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:43.259+0000 12a58640 -1 osd.6 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:44.330 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:44.325+0000 12a58640 -1 osd.6 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:45.452 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:45.442+0000 12a58640 -1 osd.6 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:46.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:46.286+0000 12a58640 -1 osd.1 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:4 32.1b 32:d96a01e0:::testobj:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-27T03:45:46.726 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:46.605+0000 12a58640 -1 osd.6 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:47.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:47.542+0000 12a58640 -1 osd.1 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:4 32.1b 32:d96a01e0:::testobj:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-27T03:45:48.075 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:48.055+0000 12a58640 -1 osd.6 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:48.293 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-01-27T03:45:48.293 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] Global test environment set-up. 2024-01-27T03:45:48.293 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-01-27T03:45:48.293 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-01-27T03:45:48.293 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-01-27T03:45:48.293 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-01-27T03:45:48.293 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: 2024-01-27T03:45:48.293 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: 2024-01-27T03:44:49.590+0000 7fcaa6c42780 -1 failed for service _ceph-mon._tcp 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: 2024-01-27T03:44:49.590+0000 7fcaa6c42780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (192 ms) 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2021 ms) 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2213 ms total) 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-01-27T03:45:48.294 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df12fcb0 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1fa2a0 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1aabc0 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6defa9460 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfbc7af0 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df319850 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df3240c0 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df324a70 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df3253e0 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca3ba0 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1f5400 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca4b80 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca3940 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca5a50 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca4930 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca69b0 2024-01-27T03:45:48.295 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca5780 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca7870 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca65f0 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df326370 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df326d40 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca7570 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0faf20 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fb9b0 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df326b10 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fc850 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fb720 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fd6f0 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fc600 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fe5b0 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fd480 2024-01-27T03:45:48.296 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0ff430 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fe350 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1002d0 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0ff1b0 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df100dc0 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df101800 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df100060 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1026c0 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1015d0 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df103550 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df102430 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1043f0 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df103310 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df105290 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df104170 2024-01-27T03:45:48.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df106100 2024-01-27T03:45:48.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df105030 2024-01-27T03:45:48.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df106c90 2024-01-27T03:45:48.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df107700 2024-01-27T03:45:48.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df105ed0 2024-01-27T03:45:48.517 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:48.510+0000 12a58640 -1 osd.1 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:4 32.1b 32:d96a01e0:::testobj:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-27T03:45:49.215 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:49.192+0000 12a58640 -1 osd.6 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:49.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:49.655+0000 12a58640 -1 osd.1 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:4 32.1b 32:d96a01e0:::testobj:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-27T03:45:49.815 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: starte api_pool: Running main() from gmock_main.cc 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [----------] Global test environment set-up. 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ OK ] LibRadosPools.PoolList (7007 ms) 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (6406 ms) 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (5932 ms) 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3808 ms) 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (6351 ms) 2024-01-27T03:45:49.816 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5891 ms) 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6108 ms) 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (8301 ms) 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10568 ms) 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [----------] 9 tests from LibRadosPools (60372 ms total) 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [----------] Global test environment tear-down 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (60372 ms total) 2024-01-27T03:45:49.817 INFO:tasks.workunit.client.0.smithi110.stdout: api_pool: [ PASSED ] 9 tests. 2024-01-27T03:45:50.248 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:50.243+0000 12a58640 -1 osd.6 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:51.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:51.047+0000 12a58640 -1 osd.1 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:4 32.1b 32:d96a01e0:::testobj:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-27T03:45:51.542 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:51.530+0000 12a58640 -1 osd.6 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:52.654 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:52.635+0000 12a58640 -1 osd.1 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:4 32.1b 32:d96a01e0:::testobj:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-27T03:45:53.105 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:53.066+0000 12a58640 -1 osd.6 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:54.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:54.034+0000 12a58640 -1 osd.1 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:4 32.1b 32:d96a01e0:::testobj:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-27T03:45:54.258 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:54.238+0000 12a58640 -1 osd.6 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:55.302 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:41.258 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:55.373 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:41.329 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:55.455 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:03:41.401 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:55.461 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:03:41.407 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:55.462 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:41.418 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:55.536 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:41.492 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:55.553 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:55.523+0000 12a58640 -1 osd.6 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:55.614 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:03:41.559 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:55.658 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:03:41.614 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:55.715 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:03:41.672 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:45:56.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:45:56.145+0000 ea50640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4634.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-27T03:45:56.542 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:56.531+0000 12a58640 -1 osd.6 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:57.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:45:57.401+0000 ea50640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4634.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-27T03:45:57.566 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: Running main() from gmock_main.cc 2024-01-27T03:45:57.566 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-01-27T03:45:57.566 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [----------] Global test environment set-up. 2024-01-27T03:45:57.566 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-01-27T03:45:57.566 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-01-27T03:45:57.566 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: handle_notify cookie 94870915648144 notify_id 219043332096 notifier_gid 4507 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (39960 ms) 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: Trying... 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: handle_notify cookie 94870927901552 notify_id 287762808832 notifier_gid 4714 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: Waiting for 3.000000000s 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: Timed out. 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: Flushing... 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: Flushed... 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (27754 ms) 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (67714 ms total) 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [----------] Global test environment tear-down 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (67714 ms total) 2024-01-27T03:45:57.567 INFO:tasks.workunit.client.0.smithi110.stdout: watch_notify: [ PASSED ] 2 tests. 2024-01-27T03:45:57.635 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:57.629+0000 12a58640 -1 osd.6 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:58.506 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:58.447+0000 12a58640 -1 osd.1 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:4 32.1b 32:d96a01e0:::testobj:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-27T03:45:58.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:45:58.581+0000 12a58640 -1 osd.2 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:45:58.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:45:58.700+0000 ea50640 -1 osd.0 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4634.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-27T03:45:58.882 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:45:58.875+0000 12a58640 -1 osd.6 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:45:59.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:45:59.644+0000 12a58640 -1 osd.2 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:45:59.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:45:59.709+0000 12a58640 -1 osd.1 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:4 32.1b 32:d96a01e0:::testobj:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) 2024-01-27T03:46:00.473 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:00.468+0000 12a58640 -1 osd.6 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:00.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:00.665+0000 12a58640 -1 osd.2 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:01.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:01.748+0000 12a58640 -1 osd.2 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:01.865 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:01.858+0000 12a58640 -1 osd.6 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:02.980 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:02.893+0000 12a58640 -1 osd.6 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:03.189 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:03.159+0000 12a58640 -1 osd.2 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:04.122 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:04.111+0000 12a58640 -1 osd.6 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:04.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:04.598+0000 12a58640 -1 osd.2 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:05.233 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:05.160+0000 12a58640 -1 osd.6 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:05.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:05.675+0000 12a58640 -1 osd.2 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:06.020 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:05.834+0000 ea50640 -1 osd.0 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:06.494 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:06.480+0000 12a58640 -1 osd.6 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:06.755 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:06.730+0000 12a58640 -1 osd.2 82 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:07.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:06.883+0000 ea50640 -1 osd.0 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:07.664 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:07.513+0000 12a58640 -1 osd.6 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:07.964 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:07.856+0000 12a58640 -1 osd.2 83 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:08.053 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:08.027+0000 ea50640 -1 osd.0 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:08.569 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:08.557+0000 12a58640 -1 osd.6 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:09.296 INFO:tasks.workunit.client.0.smithi110.stdout:d 0x55c6df108570 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1074b0 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca3a50 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fe220 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca7680 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fb830 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fd590 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df100170 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df102540 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca3820 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1016e0 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df107180 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df324f60 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df107bf0 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df1080e0 2024-01-27T03:46:09.297 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df106610 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df103420 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df100790 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0fc710 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca6e50 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6dfca7980 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 0x55c6df0ff2c0 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: started 73 aios 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df12fcb0 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1fa2a0 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1aabc0 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6defa9460 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfbc7af0 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df319850 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df3240c0 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df324a70 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df3253e0 2024-01-27T03:46:09.298 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca3ba0 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1f5400 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca4b80 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca3940 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca5a50 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca4930 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca69b0 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca5780 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca7870 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca65f0 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df326370 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df326d40 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca7570 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0faf20 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fb9b0 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df326b10 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fc850 2024-01-27T03:46:09.299 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fb720 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fd6f0 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fc600 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fe5b0 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fd480 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0ff430 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fe350 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1002d0 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0ff1b0 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df100dc0 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df101800 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df100060 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1026c0 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1015d0 2024-01-27T03:46:09.300 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df103550 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df102430 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1043f0 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df103310 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df105290 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df104170 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df106100 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df105030 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df106c90 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df107700 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df105ed0 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df108570 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1074b0 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca3a50 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fe220 2024-01-27T03:46:09.301 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca7680 2024-01-27T03:46:09.302 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fb830 2024-01-27T03:46:09.302 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fd590 2024-01-27T03:46:09.302 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df100170 2024-01-27T03:46:09.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:09.277+0000 ea50640 -1 osd.0 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:09.546 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:09.540+0000 12a58640 -1 osd.6 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:09.583 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:09.437+0000 12a58640 -1 osd.2 85 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:10.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:10.621+0000 12a58640 -1 osd.2 86 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:10.656 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:10.646+0000 12a58640 -1 osd.6 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39.616a4637 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:10.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:10.645+0000 ea50640 -1 osd.0 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:11.726 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:11.716+0000 12a58640 -1 osd.2 87 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:11.862 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:11.854+0000 ea50640 -1 osd.0 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:12.246 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:12.066+0000 12a58640 -1 osd.6 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39:ec625686:test-rados-api-smithi110-40595-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:12.874 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:12.859+0000 ea50640 -1 osd.0 87 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:12.888 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:12.796+0000 12a58640 -1 osd.2 88 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:13.665 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:13.549+0000 12a58640 -1 osd.6 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39:ec625686:test-rados-api-smithi110-40595-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:13.854 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:13.847+0000 ea50640 -1 osd.0 87 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:13.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:13.979+0000 12a58640 -1 osd.2 88 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:14.788 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:14.772+0000 12a58640 -1 osd.6 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39:ec625686:test-rados-api-smithi110-40595-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:14.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:14.951+0000 ea50640 -1 osd.0 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:15.484 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T03:46:15.397+0000 ea50640 -1 osd.7 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4528.0:8 28.2 28:452ae4ca:::6.obj:head [write 0~1 in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62) 2024-01-27T03:46:15.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:15.574+0000 12a58640 -1 osd.2 90 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4696.0:2 66.0 66.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53) 2024-01-27T03:46:15.891 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:15.812+0000 12a58640 -1 osd.6 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39:ec625686:test-rados-api-smithi110-40595-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:16.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:16.304+0000 ea50640 -1 osd.0 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:16.669 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T03:46:16.652+0000 ea50640 -1 osd.7 91 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4528.0:8 28.2 28:452ae4ca:::6.obj:head [write 0~1 in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62) 2024-01-27T03:46:16.918 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:16.912+0000 12a58640 -1 osd.6 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39:ec625686:test-rados-api-smithi110-40595-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:17.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:17.369+0000 ea50640 -1 osd.0 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:18.370 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:18.255+0000 12a58640 -1 osd.6 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39:ec625686:test-rados-api-smithi110-40595-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:18.618 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:18.548+0000 ea50640 -1 osd.0 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:19.309 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:19.302+0000 12a58640 -1 osd.6 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39:ec625686:test-rados-api-smithi110-40595-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:20.888 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:20.882+0000 12a58640 -1 osd.6 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.7s0 39:ec625686:test-rados-api-smithi110-40595-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-27T03:46:21.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:21.132+0000 ea50640 -1 osd.0 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:21.705 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:07.661 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:46:21.711 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:07.667 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:46:21.752 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:07.709 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:46:21.758 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:07.714 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:46:21.807 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:04:07.753 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:46:21.809 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:04:07.755 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:46:21.943 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:07.899 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:46:21.994 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:04:07.940 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:46:22.081 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:08.037 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:46:22.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:22.240+0000 ea50640 -1 osd.0 96 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:23.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:23.322+0000 ea50640 -1 osd.0 96 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:24.583 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:24.559+0000 ea50640 -1 osd.0 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:25.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:25.659+0000 ea50640 -1 osd.0 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:26.987 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:26.865+0000 ea50640 -1 osd.0 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:28.551 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:28.451+0000 ea50640 -1 osd.0 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:29.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:29.586+0000 ea50640 -1 osd.0 102 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:30.501 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:30.382+0000 ea50640 -1 osd.5 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:30.653 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:30.609+0000 ea50640 -1 osd.0 103 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:31.770 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:31.738+0000 ea50640 -1 osd.5 105 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:32.204 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:32.036+0000 ea50640 -1 osd.0 104 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:33.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:33.242+0000 ea50640 -1 osd.0 106 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:33.531 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:33.501+0000 ea50640 -1 osd.5 106 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:34.514 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:34.424+0000 ea50640 -1 osd.0 106 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:34.811 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:34.733+0000 ea50640 -1 osd.5 108 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:35.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:35.437+0000 ea50640 -1 osd.0 107 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:36.313 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:36.293+0000 ea50640 -1 osd.5 109 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:37.038 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:36.984+0000 ea50640 -1 osd.0 110 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:37.535 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:37.530+0000 ea50640 -1 osd.5 110 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:38.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:38.378+0000 ea50640 -1 osd.0 110 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:38.985 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:38.970+0000 ea50640 -1 osd.5 112 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:39.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:39.467+0000 ea50640 -1 osd.0 111 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:39.970 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:39.958+0000 ea50640 -1 osd.5 112 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:40.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:40.534+0000 ea50640 -1 osd.0 112 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:41.147 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:41.009+0000 ea50640 -1 osd.5 114 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:41.742 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:41.630+0000 ea50640 -1 osd.0 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:42.079 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:42.071+0000 ea50640 -1 osd.5 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:42.876 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:42.697+0000 ea50640 -1 osd.0 114 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:43.207 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:43.198+0000 ea50640 -1 osd.5 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:43.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:43.753+0000 ea50640 -1 osd.0 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:44.375 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:44.287+0000 ea50640 -1 osd.5 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:44.888 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:44.864+0000 ea50640 -1 osd.0 116 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:45.816 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:45.705+0000 ea50640 -1 osd.5 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:46.081 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:45.967+0000 ea50640 -1 osd.0 117 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4462.0:4 19.14 19:2ea7189f:test-rados-api-smithi110-40364-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) 2024-01-27T03:46:46.900 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:46.890+0000 ea50640 -1 osd.5 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:48.568 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:48.513+0000 ea50640 -1 osd.5 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (119632 ms) 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (119632 ms total) 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_cls_remote_reads: 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (119633 ms total) 2024-01-27T03:46:49.283 INFO:tasks.workunit.client.0.smithi110.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-01-27T03:46:49.748 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:49.740+0000 ea50640 -1 osd.5 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:50.093 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:50.083+0000 ea50640 -1 osd.0 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4426.0:10 18.f 18:f4d746bd:test-rados-api-smithi110-40252-6::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-01-27T03:46:50.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:50.414+0000 12a58640 -1 osd.2 122 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:46:49.440170+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:46:50.734 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:50.729+0000 ea50640 -1 osd.5 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:51.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:51.340+0000 ea50640 -1 osd.0 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4426.0:10 18.f 18:f4d746bd:test-rados-api-smithi110-40252-6::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-01-27T03:46:51.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:51.594+0000 12a58640 -1 osd.2 124 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:46:49.440170+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:46:51.985 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:51.907+0000 ea50640 -1 osd.5 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:52.535 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:52.431+0000 ea50640 -1 osd.0 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4426.0:10 18.f 18:f4d746bd:test-rados-api-smithi110-40252-6::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-01-27T03:46:53.034 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:53.031+0000 12a58640 -1 osd.2 126 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:46:49.440170+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:46:53.213 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:53.165+0000 ea50640 -1 osd.5 125 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:53.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:46:53.433+0000 ea50640 -1 osd.0 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4426.0:10 18.f 18:f4d746bd:test-rados-api-smithi110-40252-6::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-01-27T03:46:54.191 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:54.162+0000 12a58640 -1 osd.2 126 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:46:54.128395+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:46:54.211 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:54.196+0000 ea50640 -1 osd.5 125 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:55.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:55.394+0000 12a58640 -1 osd.2 127 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:46:54.128395+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:46:56.646 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:56.638+0000 ea50640 -1 osd.5 127 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:56.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:56.537+0000 12a58640 -1 osd.2 128 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:46:55.684594+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:46:56.874 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:56.784+0000 12a58640 -1 osd.6 126 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:46:57.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:57.674+0000 12a58640 -1 osd.2 129 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:46:55.684594+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:46:57.996 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:57.887+0000 12a58640 -1 osd.6 127 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:46:58.043 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:46:57.964+0000 12a58640 -1 osd.2 129 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:7 33.d 33:b46e3eb9:::5.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-01-27T03:46:58.533 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:58.519+0000 ea50640 -1 osd.5 129 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:46:59.049 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:46:58.942+0000 12a58640 -1 osd.6 128 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:46:59.612 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:46:59.602+0000 ea50640 -1 osd.5 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:47:00.483 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:47:00.386+0000 12a58640 -1 osd.6 129 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:47:00.756 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:00.586+0000 12a58640 -1 osd.2 130 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:46:55.684594+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:00.757 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:47:00.724+0000 ea50640 -1 osd.5 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:47:01.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:00.630+0000 12a58640 -1 osd.2 130 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4448.0:7 33.d 33:b46e3eb9:::5.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-01-27T03:47:01.695 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:47:01.610+0000 12a58640 -1 osd.6 130 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:47:02.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:02.239+0000 12a58640 -1 osd.2 130 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:46:55.684594+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:02.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:02.247+0000 12a58640 -1 osd.2 130 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4448.0:7 33.d 33:b46e3eb9:::5.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-01-27T03:47:02.626 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:47:02.614+0000 ea50640 -1 osd.5 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:47:02.989 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:47:02.965+0000 12a58640 -1 osd.6 132 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:47:03.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:03.488+0000 12a58640 -1 osd.2 132 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:02.279896+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:03.757 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:03.752+0000 12a58640 -1 osd.2 132 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4448.0:7 33.d 33:b46e3eb9:::5.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-01-27T03:47:03.828 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:47:03.759+0000 ea50640 -1 osd.5 134 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:47:04.023 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:47:04.013+0000 12a58640 -1 osd.6 133 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:47:04.915 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:47:04.905+0000 ea50640 -1 osd.5 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:47:04.947 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:04.853+0000 12a58640 -1 osd.2 133 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:02.279896+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:05.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:05.096+0000 12a58640 -1 osd.2 133 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4448.0:7 33.d 33:b46e3eb9:::5.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-01-27T03:47:05.110 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:47:05.105+0000 12a58640 -1 osd.6 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:47:05.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:47:05.112+0000 12a58640 -1 osd.1 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4750.0:2 84.6 84.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e103) 2024-01-27T03:47:05.525 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:47:05.522+0000 12a58640 -1 osd.4 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:1 88.3s0 88.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-01-27T03:47:05.924 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:47:05.915+0000 ea50640 -1 osd.5 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:47:06.075 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:06.063+0000 12a58640 -1 osd.2 134 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:05.220933+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:06.079 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:06.073+0000 12a58640 -1 osd.2 134 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4448.0:7 33.d 33:b46e3eb9:::5.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-01-27T03:47:06.387 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:47:06.197+0000 12a58640 -1 osd.6 136 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:47:06.971 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:47:06.966+0000 ea50640 -1 osd.5 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:47:07.092 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:53.048 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:07.094 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:53.050 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:07.132 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:53.088 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:07.162 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:04:53.108 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:07.176 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:04:53.122 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:07.181 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:53.137 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:07.229 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:07.110+0000 12a58640 -1 osd.2 135 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:05.220933+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:07.252 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:07.119+0000 12a58640 -1 osd.2 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:7 33.d 33:b46e3eb9:::5.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-01-27T03:47:07.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:47:07.291+0000 12a58640 -1 osd.1 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4750.0:2 84.6 84:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106) 2024-01-27T03:47:07.457 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:04:53.413 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:07.473 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:47:07.441+0000 12a58640 -1 osd.6 137 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.5.15:1 76.6 76.7fc1f406 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e78) 2024-01-27T03:47:07.540 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:04:53.485 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:07.594 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:04:53.550 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:07.951 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:47:07.937+0000 ea50640 -1 osd.5 137 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4732.0:3 77.6 77:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2024-01-27T03:47:08.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:08.258+0000 12a58640 -1 osd.2 136 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:05.220933+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:08.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:47:08.528+0000 12a58640 -1 osd.1 137 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4750.0:2 84.6 84:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106) 2024-01-27T03:47:08.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:08.560+0000 12a58640 -1 osd.2 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4448.0:7 33.d 33:b46e3eb9:::5.obj:head [write 0~168 in=168b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) 2024-01-27T03:47:09.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:09.540+0000 12a58640 -1 osd.2 137 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:08.355804+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:10.874 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:10.858+0000 12a58640 -1 osd.2 138 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:08.355804+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:12.200 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:12.146+0000 12a58640 -1 osd.2 140 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:08.355804+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:12.655 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T03:47:12.626+0000 ea50640 -1 osd.7 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4768.0:2 90.15 90:ad551702:::test:head [call hello.record_hello in=21b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) 2024-01-27T03:47:13.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:13.362+0000 12a58640 -1 osd.2 140 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:08.355804+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:13.621 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T03:47:13.611+0000 ea50640 -1 osd.7 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4768.0:2 90.15 90:ad551702:::test:head [call hello.record_hello in=21b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) 2024-01-27T03:47:14.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:14.453+0000 12a58640 -1 osd.2 141 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:14.035670+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:14.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:14.645+0000 12a58640 -1 osd.2 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4685.0:2 81.0 81:0384ab8e:::oid:head [writefull 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) 2024-01-27T03:47:14.736 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T03:47:14.693+0000 ea50640 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4768.0:2 90.15 90:ad551702:::test:head [call hello.record_hello in=21b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) 2024-01-27T03:47:15.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:15.614+0000 12a58640 -1 osd.2 143 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:14.035670+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:16.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:16.980+0000 12a58640 -1 osd.2 144 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:15.779095+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:18.179 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:18.152+0000 12a58640 -1 osd.2 144 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:15.779095+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:19.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:19.456+0000 12a58640 -1 osd.2 145 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:15.779095+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:20.765 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:20.667+0000 12a58640 -1 osd.2 146 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:19.817732+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:21.860 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:21.857+0000 12a58640 -1 osd.2 147 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:19.817732+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:23.214 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:23.102+0000 12a58640 -1 osd.2 148 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:19.817732+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:23.322 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:47:23.312+0000 12a58640 -1 osd.6 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4683.0:2 82.5 82:a48cab6d:test-rados-api-smithi110-40250-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) 2024-01-27T03:47:24.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:24.191+0000 ea50640 -1 osd.0 149 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:24.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:24.641+0000 12a58640 -1 osd.2 150 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:19.817732+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:25.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:25.273+0000 ea50640 -1 osd.0 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:26.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:25.918+0000 12a58640 -1 osd.2 152 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:25.692686+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:26.538 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:26.516+0000 ea50640 -1 osd.0 151 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:27.001 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-01-27T03:47:26.987+0000 d37e640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] for 122sec e126 v126) 2024-01-27T03:47:27.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:27.321+0000 12a58640 -1 osd.2 153 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:25.692686+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:27.465 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:13.421 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:27.478 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:13.434 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:27.532 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:13.488 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:27.542 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:05:13.487 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:27.544 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:13.500 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:27.552 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:05:13.498 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:27.603 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:13.559 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:27.656 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:05:13.602 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:27.663 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:13.618 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:27.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:27.770+0000 ea50640 -1 osd.0 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:28.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:28.351+0000 12a58640 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:25.692686+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:28.875 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:28.868+0000 ea50640 -1 osd.0 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:29.558 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:29.415+0000 12a58640 -1 osd.2 155 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:25.692686+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:30.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:30.143+0000 ea50640 -1 osd.0 155 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:30.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:30.821+0000 12a58640 -1 osd.2 156 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:30.677488+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:31.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:31.261+0000 ea50640 -1 osd.0 156 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:32.004 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:31.983+0000 12a58640 -1 osd.2 157 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:30.677488+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:32.088 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-01-27T03:47:32.086+0000 d37e640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] for 122sec e126 v126) 2024-01-27T03:47:32.719 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:32.546+0000 ea50640 -1 osd.0 158 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:33.225 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:33.119+0000 12a58640 -1 osd.2 158 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:30.677488+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:34.768 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:34.754+0000 ea50640 -1 osd.0 159 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:35.071 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:35.025+0000 12a58640 -1 osd.2 160 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:33.769302+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:36.212 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:36.144+0000 ea50640 -1 osd.0 161 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:36.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:36.211+0000 12a58640 -1 osd.2 161 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:33.769302+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:37.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-01-27T03:47:37.124+0000 d37e640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] for 122sec e126 v126) 2024-01-27T03:47:37.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:37.300+0000 ea50640 -1 osd.0 163 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:37.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:37.322+0000 12a58640 -1 osd.2 162 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:33.769302+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:38.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:38.479+0000 12a58640 -1 osd.2 163 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:33.769302+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:38.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:38.589+0000 ea50640 -1 osd.0 163 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:39.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:39.830+0000 12a58640 -1 osd.2 164 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:38.589528+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:40.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:40.601+0000 ea50640 -1 osd.0 165 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:41.610 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:41.599+0000 ea50640 -1 osd.0 165 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:41.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:41.547+0000 12a58640 -1 osd.2 165 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:38.589528+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:42.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-01-27T03:47:42.294+0000 d37e640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] for 122sec e126 v126) 2024-01-27T03:47:42.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:42.682+0000 ea50640 -1 osd.0 167 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:43.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:43.499+0000 12a58640 -1 osd.2 167 heartbeat_check: no reply from 172.21.15.162:6806 osd.5 since back 2024-01-27T03:47:38.589528+0000 front 2024-01-27T03:44:51.818218+0000 (oldest deadline 2024-01-27T03:46:49.094342+0000) 2024-01-27T03:47:43.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:43.764+0000 ea50640 -1 osd.0 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:45.344 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:45.295+0000 ea50640 -1 osd.0 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:45.790 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:45.734+0000 12a58640 -1 osd.2 170 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:47:46.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:46.398+0000 ea50640 -1 osd.0 171 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:47.540 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:47.455+0000 ea50640 -1 osd.0 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:47.953 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:47.944+0000 12a58640 -1 osd.2 171 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:47:48.976 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:48.861+0000 ea50640 -1 osd.0 174 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:49.040 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:49.023+0000 12a58640 -1 osd.2 173 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:47:50.072 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:50.060+0000 12a58640 -1 osd.2 174 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:47:50.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:50.102+0000 ea50640 -1 osd.0 175 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:51.197 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:51.193+0000 ea50640 -1 osd.0 176 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:51.201 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:51.061+0000 12a58640 -1 osd.2 175 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:47:52.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:52.140+0000 12a58640 -1 osd.2 176 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4406.0:13 31.13 31:cd8c8e4c:::foo2:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) 2024-01-27T03:47:52.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:52.310+0000 ea50640 -1 osd.0 176 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:52.459 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:38.415 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:52.492 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:38.448 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:52.520 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:38.476 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:52.539 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:38.495 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:52.543 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:05:38.488 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:52.551 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:05:38.497 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:52.664 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:05:38.620 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:52.695 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:05:38.641 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:52.703 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:05:38.659 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:47:53.244 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:53.236+0000 12a58640 -1 osd.2 177 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:47:53.388 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:53.380+0000 ea50640 -1 osd.0 177 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:54.374 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:54.362+0000 12a58640 -1 osd.2 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:47:54.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:54.386+0000 ea50640 -1 osd.0 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:55.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:55.421+0000 ea50640 -1 osd.0 179 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:55.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:55.417+0000 12a58640 -1 osd.2 179 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:47:56.545 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:56.451+0000 ea50640 -1 osd.0 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:56.555 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:47:56.459+0000 12a58640 -1 osd.2 181 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:47:57.467 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:57.455+0000 ea50640 -1 osd.0 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:58.545 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:47:58.537+0000 ea50640 -1 osd.0 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:47:59.793 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:47:59.777+0000 12a58640 -1 osd.4 182 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:00.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:00.066+0000 ea50640 -1 osd.0 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:00.903 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:00.892+0000 12a58640 -1 osd.4 183 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:01.779 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:01.752+0000 12a58640 -1 osd.2 185 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4406.0:13 31.13 31:cd8c8e4c:::foo2:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) 2024-01-27T03:48:02.224 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:02.048+0000 12a58640 -1 osd.4 185 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:02.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:02.423+0000 ea50640 -1 osd.0 186 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:03.712 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:03.592+0000 ea50640 -1 osd.0 187 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:03.717 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:03.619+0000 12a58640 -1 osd.4 186 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:04.174 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:04.073+0000 12a58640 -1 osd.2 188 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4406.0:13 31.13 31:cd8c8e4c:::foo2:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) 2024-01-27T03:48:04.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:04.660+0000 ea50640 -1 osd.0 188 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:05.148 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:05.040+0000 12a58640 -1 osd.4 188 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:05.219 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:05.203+0000 12a58640 -1 osd.2 189 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:48:05.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:05.678+0000 ea50640 -1 osd.0 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:06.238 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:06.184+0000 12a58640 -1 osd.4 188 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:06.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:06.339+0000 12a58640 -1 osd.2 190 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:48:07.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:07.025+0000 ea50640 -1 osd.0 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:07.410 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:48:07.397+0000 12a58640 -1 osd.6 190 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4494.0:49 12.6 12:63e36dab:test-rados-api-smithi110-40874-6::testobj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151) 2024-01-27T03:48:07.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:07.383+0000 12a58640 -1 osd.2 191 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:48:07.455 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:07.445+0000 12a58640 -1 osd.4 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:08.522 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:08.404+0000 ea50640 -1 osd.0 192 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:08.588 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:08.575+0000 12a58640 -1 osd.4 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:08.989 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:08.958+0000 12a58640 -1 osd.2 192 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:48:09.015 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:48:09.003+0000 12a58640 -1 osd.6 190 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4494.0:49 12.6 12:63e36dab:test-rados-api-smithi110-40874-6::testobj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151) 2024-01-27T03:48:09.655 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:09.644+0000 12a58640 -1 osd.4 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:09.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:09.529+0000 ea50640 -1 osd.0 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:10.216 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:48:10.111+0000 12a58640 -1 osd.6 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 101.0 101.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e163) 2024-01-27T03:48:10.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:10.614+0000 ea50640 -1 osd.0 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:10.856 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:10.550+0000 12a58640 -1 osd.2 193 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:48:10.882 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:10.872+0000 12a58640 -1 osd.4 192 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:11.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:11.773+0000 12a58640 -1 osd.2 194 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4426.0:13 18.1f 18:fe8ebdf4:test-rados-api-smithi110-40252-7::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135) 2024-01-27T03:48:11.855 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:48:11.819+0000 12a58640 -1 osd.6 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 101.0 101.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e163) 2024-01-27T03:48:12.014 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:11.799+0000 ea50640 -1 osd.0 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:12.241 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:12.161+0000 12a58640 -1 osd.4 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:13.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:12.993+0000 12a58640 -1 osd.2 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4730.0:2 93.1b 93.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e141) 2024-01-27T03:48:13.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:12.950+0000 ea50640 -1 osd.0 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:13.216 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:13.210+0000 12a58640 -1 osd.4 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94.f14f953a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:13.217 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:48:13.193+0000 12a58640 -1 osd.6 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 101.0 101.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e163) 2024-01-27T03:48:13.498 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:48:13.492+0000 12a58640 -1 osd.1 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:14 26.3 26:c5d00cd9:::12.obj:head [write 0~79 in=79b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165) 2024-01-27T03:48:14.017 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T03:48:14.008+0000 ea50640 -1 osd.0 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4708.0:5 68.2 68:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-27T03:48:14.204 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:48:14.179+0000 12a58640 -1 osd.6 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 101.0 101.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e163) 2024-01-27T03:48:14.215 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:48:14.200+0000 ea50640 -1 osd.5 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4616.0:38 58.15 58:af44a1d6:::obj35:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e166) 2024-01-27T03:48:14.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T03:48:14.296+0000 12a58640 -1 osd.2 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4730.0:2 93.1b 93.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e141) 2024-01-27T03:48:14.349 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T03:48:14.334+0000 12a58640 -1 osd.4 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4777.0:3 94.1a 94:5ca9f28f:test-rados-api-smithi110-40481-8::foo:head [write 0~128 in=128b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e145) 2024-01-27T03:48:14.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T03:48:14.550+0000 12a58640 -1 osd.1 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:14 26.3 26:c5d00cd9:::12.obj:head [write 0~79 in=79b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165) 2024-01-27T03:48:15.242 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:48:15.226+0000 12a58640 -1 osd.6 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 101.0 101.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e163) 2024-01-27T03:48:15.303 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T03:48:15.287+0000 ea50640 -1 osd.5 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4616.0:38 58.15 58:af44a1d6:::obj35:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e166) 2024-01-27T03:48:16.531 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:48:16.491+0000 12a58640 -1 osd.6 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 101.0 101.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e163) 2024-01-27T03:48:17.573 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T03:48:17.563+0000 12a58640 -1 osd.6 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4813.0:2 101.0 101:0384ab8e:::oid:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163) 2024-01-27T03:48:24.628 INFO:tasks.workunit.client.0.smithi110.stdout: cls: Running main() from gmock_main.cc 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [----------] Global test environment set-up. 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [ OK ] NeoRadosCls.DNE (37397 ms) 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (177499 ms) 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [----------] 2 tests from NeoRadosCls (214896 ms total) 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [----------] Global test environment tear-down 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [==========] 2 tests from 1 test suite ran. (214896 ms total) 2024-01-27T03:48:24.629 INFO:tasks.workunit.client.0.smithi110.stdout: cls: [ PASSED ] 2 tests. 2024-01-27T03:48:27.791 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-01-27T03:48:27.791 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-01-27T03:48:27.791 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-01-27T03:48:27.791 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-01-27T03:48:27.791 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (576 ms) 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (434 ms) 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (217222 ms) 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (218232 ms total) 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (218232 ms total) 2024-01-27T03:48:27.792 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-01-27T03:48:28.048 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: Running main() from gmock_main.cc 2024-01-27T03:48:28.048 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-01-27T03:48:28.048 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [----------] Global test environment set-up. 2024-01-27T03:48:28.048 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6747 ms) 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (14866 ms) 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (196695 ms) 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [----------] 3 tests from NeoRadosCmd (218308 ms total) 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [----------] Global test environment tear-down 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [==========] 3 tests from 1 test suite ran. (218308 ms total) 2024-01-27T03:48:28.049 INFO:tasks.workunit.client.0.smithi110.stdout: cmd: [ PASSED ] 3 tests. 2024-01-27T03:48:30.913 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-01-27T03:48:30.913 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-01-27T03:48:30.913 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-01-27T03:48:30.913 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-01-27T03:48:30.913 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (44112 ms) 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (42966 ms) 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (4804 ms) 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (100781 ms) 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (5340 ms) 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (934 ms) 2024-01-27T03:48:30.914 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (573 ms) 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (2344 ms) 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (124 ms) 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (129 ms) 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (223 ms) 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (265 ms) 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-01-27T03:48:30.915 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (2021 ms) 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (1760 ms) 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (273 ms) 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (128 ms) 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (206778 ms total) 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (221263 ms total) 2024-01-27T03:48:30.916 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-01-27T03:48:30.962 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:16.917 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:48:30.973 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:16.928 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:48:31.317 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:06:17.273 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:48:31.527 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:17.483 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:48:31.590 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:17.545 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:48:31.969 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:06:17.925 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:48:31.987 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:06:17.933 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:48:32.002 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:06:17.948 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:48:32.165 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:06:18.111 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:48:37.822 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: Running main() from gmock_main.cc 2024-01-27T03:48:37.822 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-01-27T03:48:37.822 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [----------] Global test environment set-up. 2024-01-27T03:48:37.822 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-01-27T03:48:37.822 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-01-27T03:48:37.822 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1256 ms) 2024-01-27T03:48:37.822 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-01-27T03:48:37.822 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (316 ms) 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (216583 ms) 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:23.883906+0000 mon.a [INF] from='client.? 172.21.15.110:0/188531446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi110-40595-1"}]: dispatch 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:24.288462+0000 mon.b [INF] from='client.? 172.21.15.110:0/1316496017' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi110-40469-6", "overlaypool": "test-rados-api-smithi110-40469-7"}]: dispatch 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:25.324300+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi110-40469-6", "overlaypool": "test-rados-api-smithi110-40469-7"}]: dispatch 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:25.508995+0000 mon.a [WRN] Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:25.509436+0000 mon.a [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 1 pg peering) 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:25.510037+0000 mon.a [INF] Health check cleared: SLOW_OPS (was: 5 slow ops, oldest one blocked for 96 sec, daemons [osd.2,osd.4,osd.5,osd.6] have slow ops.) 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:27.136712+0000 mon.b [INF] from='client.? 172.21.15.110:0/3996048310' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi110-40530-7"}]: dispatch 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:27.285195+0000 mon.a [INF] from='client.? 172.21.15.110:0/2889209441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi110-40221-10"}]: dispatch 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:29.071250+0000 mon.a [INF] from='client.? 172.21.15.110:0/188531446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi110-40595-1"}]': finished 2024-01-27T03:48:37.823 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:29.081646+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi110-40469-6", "overlaypool": "test-rados-api-smithi110-40469-7"}]': finished 2024-01-27T03:48:37.824 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:29.532435+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi110-40530-7"}]: dispatch 2024-01-27T03:48:37.824 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:29.858149+0000 mon.a [INF] from='client.? 172.21.15.110:0/188531446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi110-40595-1"}]: dispatch 2024-01-27T03:48:37.824 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:30.384184+0000 mon.a [INF] from='client.? 172.21.15.110:0/1808582427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi110-40169-16"}]: dispatch 2024-01-27T03:48:37.824 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:30.673723+0000 client.admin [INF] onexx 2024-01-27T03:48:39.011 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:30.692307+0000 mon.a [INF] from='client.? 172.21.15.110:0/1963524205' entity='client.admin' cmd=[{"pref api_lock_pp: Running main() from gmock_main.cc 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: seed 40221 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (31797 ms) 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (16573 ms) 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-01-27T03:48:39.012 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (3158 ms) 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (24134 ms) 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (4795 ms) 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (8649 ms) 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (46900 ms) 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (17861 ms) 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (153867 ms total) 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-01-27T03:48:39.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (16962 ms) 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (6135 ms) 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (22381 ms) 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1190 ms) 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (1586 ms) 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (464 ms) 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (1620 ms) 2024-01-27T03:48:39.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-01-27T03:48:39.015 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (371 ms) 2024-01-27T03:48:39.015 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (50709 ms total) 2024-01-27T03:48:39.015 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: 2024-01-27T03:48:39.015 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-01-27T03:48:39.015 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (229639 ms total) 2024-01-27T03:48:39.015 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-01-27T03:48:43.990 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-01-27T03:48:43.990 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [----------] Global test environment set-up. 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [----------] 7 tests from LibRadosList 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosList.ListObjects (131714 ms) 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (69816 ms) 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: myset foo1,foo2,foo3 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo1 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo2 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo3 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: myset foo1,foo4,foo5 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo4 2024-01-27T03:48:43.991 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo5 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo1 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: myset foo6,foo7 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo7 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo6 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns1:foo4 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns1:foo5 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns2:foo7 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns2:foo6 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns1:foo1 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: :foo1 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: :foo2 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: :foo3 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (14490 ms) 2024-01-27T03:48:43.992 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 1 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 10 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 13 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 7 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 14 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 0 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 15 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 11 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 5 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 8 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 6 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 3 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 4 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 12 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 9 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 2 0 2024-01-27T03:48:43.993 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (7541 ms) 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: x cursor=MIN 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=1 cursor=16:02547ec2:::1:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=10 cursor=16:52ea6a34:::10:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=13 cursor=16:566253c9:::13:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=7 cursor=16:5c6b0b28:::7:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=14 cursor=16:62a1935d:::14:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=0 cursor=16:6cac518f:::0:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=15 cursor=16:863748b0:::15:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=11 cursor=16:89d3ae78:::11:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=5 cursor=16:b29083e3:::5:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=8 cursor=16:bd63b0f1:::8:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=6 cursor=16:c4fdafeb:::6:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=3 cursor=16:cfc208b3:::3:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=4 cursor=16:d83876eb:::4:head 2024-01-27T03:48:43.994 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=12 cursor=16:de5d7c5f:::12:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=9 cursor=16:e960b815:::9:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > oid=2 cursor=16:f905c69b:::2:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: FIRST> seek to MIN oid=1 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=1 cursor=16:02547ec2:::1:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:02547ec2:::1:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:02547ec2:::1:head -> 1 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=10 cursor=16:52ea6a34:::10:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:52ea6a34:::10:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:52ea6a34:::10:head -> 10 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=13 cursor=16:566253c9:::13:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:566253c9:::13:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:566253c9:::13:head -> 13 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=7 cursor=16:5c6b0b28:::7:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:5c6b0b28:::7:head 2024-01-27T03:48:43.995 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:5c6b0b28:::7:head -> 7 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : api_stat: Running main() from gmock_main.cc 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [----------] Global test environment set-up. 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ OK ] LibRadosStat.Stat (59776 ms) 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (13617 ms) 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-01-27T03:48:46.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ OK ] LibRadosStat.StatNS (30190 ms) 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (45 ms) 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (35464 ms) 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [----------] 5 tests from LibRadosStat (139092 ms total) 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (55454 ms) 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-01-27T03:48:46.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (3759 ms) 2024-01-27T03:48:46.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-01-27T03:48:46.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (954 ms) 2024-01-27T03:48:46.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-01-27T03:48:46.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1927 ms) 2024-01-27T03:48:46.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (62094 ms total) 2024-01-27T03:48:46.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: 2024-01-27T03:48:46.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [----------] Global test environment tear-down 2024-01-27T03:48:46.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (237168 ms total) 2024-01-27T03:48:46.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat: [ PASSED ] 9 tests. 2024-01-27T03:48:47.802 INFO:tasks.workunit.client.0.smithi110.stdout:ix":"log", "logtext":["onexx"]}]: dispatch 2024-01-27T03:48:47.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:30.695539+0000 mon.b [INF] from='client.? 172.21.15.110:0/3996048310' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi110-40530-7"}]: dispatch 2024-01-27T03:48:47.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:30.820934+0000 mon.b [INF] from='client.? 172.21.15.110:0/99113102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi110-40252-10"}]: dispatch 2024-01-27T03:48:47.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:31.316835+0000 mon.c [INF] from='client.? 172.21.15.110:0/2128776942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi110-41230-2"}]: dispatch 2024-01-27T03:48:47.802 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:31.453036+0000 mon.b [INF] from='client.4733 172.21.15.110:0/937478374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi110-40510-7"}]: dispatch 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:31.479469+0000 mon.a [INF] from='client.? 172.21.15.110:0/1808582427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi110-40169-16"}]: dispatch 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:32.360441+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi110-41230-2"}]: dispatch 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:33.016372+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi110-40530-7"}]: dispatch 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:33.028076+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi110-40252-10"}]: dispatch 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:33.040043+0000 mon.a [INF] from='client.4733 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi110-40510-7"}]: dispatch 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:33.059784+0000 mon.a [INF] from='client.? 172.21.15.110:0/2889209441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi110-40221-10"}]': finished 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:33.070143+0000 mon.a [INF] from='client.? 172.21.15.110:0/188531446' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi110-40595-1"}]': finished 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:38.056041+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 38357.749ms) (OSD_SLOW_PING_TIME_BACK) 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:38.063560+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 28058.870ms) (OSD_SLOW_PING_TIME_FRONT) 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:38.941658+0000 mon.b [INF] from='client.? 172.21.15.110:0/99113102' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi110-40252-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:39.001567+0000 client.admin [INF] twoxx 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:39.018649+0000 mon.a [INF] from='client.? 172.21.15.110:0/1963524205' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:39.070002+0000 mon.b [INF] from='client.4733 172.21.15.110:0/937478374' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi110-40510-7"}]: dispatch 2024-01-27T03:48:47.803 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:39.504658+0000 mon.c [INF] from='client.? 172.21.15.110:0/2128776942' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi110-41230-2"}]: dispatch 2024-01-27T03:48:51.385 INFO:tasks.workunit.client.0.smithi110.stdout: oid=14 cursor=16:62a1935d:::14:head 2024-01-27T03:48:51.385 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:62a1935d:::14:head 2024-01-27T03:48:51.385 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:62a1935d:::14:head -> 14 2024-01-27T03:48:51.385 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=0 cursor=16:6cac518f:::0:head 2024-01-27T03:48:51.385 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:6cac518f:::0:head 2024-01-27T03:48:51.385 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:6cac518f:::0:head -> 0 2024-01-27T03:48:51.385 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=15 cursor=16:863748b0:::15:head 2024-01-27T03:48:51.385 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:863748b0:::15:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:863748b0:::15:head -> 15 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=11 cursor=16:89d3ae78:::11:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:89d3ae78:::11:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:89d3ae78:::11:head -> 11 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=5 cursor=16:b29083e3:::5:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:b29083e3:::5:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:b29083e3:::5:head -> 5 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=8 cursor=16:bd63b0f1:::8:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:bd63b0f1:::8:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:bd63b0f1:::8:head -> 8 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=6 cursor=16:c4fdafeb:::6:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:c4fdafeb:::6:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:c4fdafeb:::6:head -> 6 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=3 cursor=16:cfc208b3:::3:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:cfc208b3:::3:head 2024-01-27T03:48:51.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:cfc208b3:::3:head -> 3 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=4 cursor=16:d83876eb:::4:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:d83876eb:::4:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:d83876eb:::4:head -> 4 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=12 cursor=16:de5d7c5f:::12:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:de5d7c5f:::12:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:de5d7c5f:::12:head -> 12 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=9 cursor=16:e960b815:::9:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:e960b815:::9:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:e960b815:::9:head -> 9 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : oid=2 cursor=16:f905c69b:::2:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:f905c69b:::2:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:f905c69b:::2:head -> 2 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:5c6b0b28:::7:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:5c6b0b28:::7:head expected=16:5c6b0b28:::7:head 2024-01-27T03:48:51.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:5c6b0b28:::7:head -> 7 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=7 expected=7 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:52ea6a34:::10:head 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:52ea6a34:::10:head expected=16:52ea6a34:::10:head 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:52ea6a34:::10:head -> 10 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=10 expected=10 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:b29083e3:::5:head 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:b29083e3:::5:head expected=16:b29083e3:::5:head 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:b29083e3:::5:head -> 5 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=5 expected=5 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:89d3ae78:::11:head 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:89d3ae78:::11:head expected=16:89d3ae78:::11:head 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:89d3ae78:::11:head -> 11 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=11 expected=11 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:863748b0:::15:head 2024-01-27T03:48:51.388 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:863748b0:::15:head expected=16:863748b0:::15:head 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:863748b0:::15:head -> 15 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=15 expected=15 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:62a1935d:::14:head 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:62a1935d:::14:head expected=16:62a1935d:::14:head 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:62a1935d:::14:head -> 14 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=14 expected=14 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:566253c9:::13:head 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:566253c9:::13:head expected=16:566253c9:::13:head 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:566253c9:::13:head -> 13 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=13 expected=13 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:02547ec2:::1:head 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:02547ec2:::1:head expected=16:02547ec2:::1:head 2024-01-27T03:48:51.389 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:02547ec2:::1:head -> 1 2024-01-27T03:48:55.470 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:39.507736+0000 mon.c [INF] from='client.? 172.21.15.110:0/666154178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi110-40187-22"}]: dispatch 2024-01-27T03:48:55.470 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:39.556720+0000 mon.b [INF] from='client.? 172.21.15.110:0/244642466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi110-40132-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-01-27T03:48:55.470 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:39.668409+0000 mon.b [INF] from='client.? 172.21.15.110:0/3984317667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi110-40783-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-01-27T03:48:55.470 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:40.762147+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi110-40252-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:40.772436+0000 mon.a [INF] from='client.4733 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi110-40510-7"}]: dispatch 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:40.805717+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi110-40132-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:40.809750+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi110-40783-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:40.981116+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi110-41230-2"}]: dispatch 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:41.007801+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi110-40187-22"}]: dispatch 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:41.032500+0000 mon.a [INF] from='client.? 172.21.15.110:0/1808582427' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi110-40169-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:41.053014+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi110-40469-6"}]': finished 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:41.075635+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi110-40530-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:41.768613+0000 mon.a [INF] from='client.? 172.21.15.110:0/1808582427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi110-40169-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi110-40169-16"}]: dispatch 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:41.871511+0000 mon.c [INF] from='client.? 172.21.15.110:0/3383809858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi110-40634-12"}]: dispatch 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:42.279044+0000 mon.a [WRN] Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-27T03:48:55.471 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:42.387515+0000 mon.b [INF] from='client.? 172.21.15.110:0/1316496017' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi110-40469-6", "tierpool": "test-rados-api-smithi110-40469-7"}]: dispatch 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_1_[41000]: starting. 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_1_[41000]: creating pool ceph_test_rados_list_parallel.smithi110-40917 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_1_[41000]: created object 0... 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_1_[41000]: created object 25... 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_1_[41000]: created object 49... 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_1_[41000]: finishing. 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_1_[41000]: shutting down. 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_2_[41001]: starting. 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_2_[41001]: listing objects. 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_2_[41001]: listed object 0... 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_2_[41001]: listed object 25... 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_2_[41001]: saw 50 objects 2024-01-27T03:49:02.861 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_2_[41001]: shutting down. 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_3_[62399]: starting. 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_3_[62399]: creating pool ceph_test_rados_list_parallel.smithi110-40917 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_3_[62399]: created object 0... 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_3_[62399]: created object 25... 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_3_[62399]: created object 49... 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_3_[62399]: finishing. 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_3_[62399]: shutting down. 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_4_[62400]: starting. 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_4_[62400]: listing objects. 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_4_[62400]: listed object 0... 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_4_[62400]: listed object 25... 2024-01-27T03:49:02.862 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_4_[62400]: saw 44 objects 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_4_[62400]: shutting down. 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_5_[62401]: starting. 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_5_[62401]: removed 25 objects... 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_5_[62401]: removed half of the objects 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_5_[62401]: removed 50 objects... 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_5_[62401]: removed 50 objects 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_5_[62401]: shutting down. 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_6_[73890]: starting. 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_6_[73890]: creating pool ceph_test_rados_list_parallel.smithi110-40917 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_6_[73890]: created object 0... 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_6_[73890]: created object 25... 2024-01-27T03:49:02.863 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_6_[73890]: created object 49... 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_6_[73890]: finishing. 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_6_[73890]: shutting down. 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_7_[73891]: starting. 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_7_[73891]: listing objects. 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_7_[73891]: listed object 0... 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_7_[73891]: listed object 25... 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_7_[73891]: listed object 50... 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_7_[73891]: saw 56 objects 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_7_[73891]: shutting down. 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_8_[73892]: starting. 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_8_[73892]: added 25 objects... 2024-01-27T03:49:02.864 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_8_[73892]: added half of the objects 2024-01-27T03:49:02.865 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_8_[73892]: added 50 objects... 2024-01-27T03:49:02.865 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_8_[73892]: added 50 objects 2024-01-27T03:49:02.865 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_8_[73892]: shutting down. 2024-01-27T03:49:02.865 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:02.865 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:05.047 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ********************** api_cmd: got: 2024-01-27T03:48:42.716518+0000 mon.b [INF] from='client.? 172.21.15.110:0/3996048310' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi110-40530-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi110-40530-7"}]: dispatch 2024-01-27T03:49:05.047 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:42.881322+0000 mon.c [INF] from='client.? 172.21.15.110:0/666154178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi110-40187-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-27T03:49:05.047 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:43.070075+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi110-40634-12"}]: dispatch 2024-01-27T03:49:05.047 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:43.161126+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi110-40252-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-01-27T03:49:05.047 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:43.177817+0000 mon.a [INF] from='client.4733 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi110-40510-7"}]': finished 2024-01-27T03:49:05.048 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:43.192268+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi110-40132-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-01-27T03:49:05.048 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:43.195493+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi110-40783-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-01-27T03:49:05.048 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:43.198568+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi110-41230-2"}]': finished 2024-01-27T03:49:05.048 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:43.450052+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi110-40469-6", "tierpool": "test-rados-api-smithi110-40469-7"}]: dispatch 2024-01-27T03:49:05.048 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:43.466703+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi110-40530-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi110-40530-7"}]: dispatch 2024-01-27T03:49:05.048 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:44.871998+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi110-40187-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-27T03:49:05.048 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:44.887337+0000 mon.a [INF] from='client.? 172.21.15.110:0/1963524205' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-01-27T03:49:05.048 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:44.254307+0000 mon.b [INF] from='client.? 172.21.15.110:0/99113102' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi110-40252-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi110-40252-10"}]: dispatch 2024-01-27T03:49:05.048 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:44.596247+0000 mon.c [INF] from='client.? 172.21.15.110:0/3383809858' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi110-40634-12"}]: dispatch 2024-01-27T03:49:15.385 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:44.819974+0000 mon.b [INF] from='client.? 172.21.15.110:0/676975287' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi110-40595-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:46.041563+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi110-40252-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi110-40252-10"}]: dispatch 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:46.095708+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi110-40595-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:46.766868+0000 mon.b [INF] from='client.? 172.21.15.110:0/1472678435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi110-41230-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:47.158324+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi110-40634-12"}]: dispatch 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:47.289707+0000 mon.a [INF] from='client.? 172.21.15.110:0/1808582427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi110-40169-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi110-40169-16"}]': finished 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:47.292992+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi110-40469-6", "tierpool": "test-rados-api-smithi110-40469-7"}]': finished 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:47.735060+0000 client.admin [INF] threexx 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:47.741960+0000 mon.a [INF] from='client.? 172.21.15.110:0/1963524205' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: got: 2024-01-27T03:48:49.177480+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi110-41230-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (47682 ms) 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (265837 ms total) 2024-01-27T03:49:15.386 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: 2024-01-27T03:49:15.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [----------] Global test environment tear-down 2024-01-27T03:49:15.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (265837 ms total) 2024-01-27T03:49:15.387 INFO:tasks.workunit.client.0.smithi110.stdout: api_cmd: [ PASSED ] 4 tests. 2024-01-27T03:49:16.834 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_1_[41013]: starting. 2024-01-27T03:49:16.834 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_1_[41013]: creating pool ceph_test_rados_open_pools_parallel.smithi110-40979 2024-01-27T03:49:16.834 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_1_[41013]: created object 0... 2024-01-27T03:49:16.834 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_1_[41013]: created object 25... 2024-01-27T03:49:16.834 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_1_[41013]: created object 49... 2024-01-27T03:49:16.834 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_1_[41013]: finishing. 2024-01-27T03:49:16.834 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_1_[41013]: shutting down. 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_2_[41016]: starting. 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_2_[41016]: rados_pool_create. 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_2_[41016]: rados_ioctx_create. 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_2_[41016]: shutting down. 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: ******************************* 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_3_[63874]: starting. 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_3_[63874]: creating pool ceph_test_rados_open_pools_parallel.smithi110-40979 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_3_[63874]: created object 0... 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_3_[63874]: created object 25... 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_3_[63874]: created object 49... 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_3_[63874]: finishing. 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_3_[63874]: shutting down. 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: ******************************* 2024-01-27T03:49:16.835 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_4_[63875]: starting. 2024-01-27T03:49:16.836 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_4_[63875]: rados_pool_create. 2024-01-27T03:49:16.836 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_4_[63875]: rados_ioctx_create. 2024-01-27T03:49:16.836 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: process_4_[63875]: shutting down. 2024-01-27T03:49:16.836 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: ******************************* 2024-01-27T03:49:16.836 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: ******************************* 2024-01-27T03:49:16.836 INFO:tasks.workunit.client.0.smithi110.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout:********* 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_9_[81311]: starting. 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_9_[81311]: creating pool ceph_test_rados_list_parallel.smithi110-40917 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_9_[81311]: created object 0... 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_9_[81311]: created object 25... 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_9_[81311]: created object 49... 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_9_[81311]: finishing. 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_9_[81311]: shutting down. 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.326 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_10_[81312]: starting. 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_10_[81312]: listing objects. 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_10_[81312]: listed object 0... 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_10_[81312]: listed object 25... 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_10_[81312]: listed object 50... 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_10_[81312]: listed object 75... 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_10_[81312]: saw 97 objects 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_10_[81312]: shutting down. 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_12_[81317]: starting. 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_12_[81317]: added 25 objects... 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_12_[81317]: added half of the objects 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_12_[81317]: added 50 objects... 2024-01-27T03:49:19.327 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_12_[81317]: added 50 objects 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_12_[81317]: shutting down. 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_13_[81318]: starting. 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_13_[81318]: removed 25 objects... 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_13_[81318]: removed half of the objects 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_13_[81318]: removed 50 objects... 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_13_[81318]: removed 50 objects 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_13_[81318]: shutting down. 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.328 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_11_[81314]: starting. 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_11_[81314]: added 25 objects... 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_11_[81314]: added half of the objects 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_11_[81314]: added 50 objects... 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_11_[81314]: added 50 objects 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_11_[81314]: shutting down. 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_14_[90882]: starting. 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_14_[90882]: creating pool ceph_test_rados_list_parallel.smithi110-40917 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_14_[90882]: created object 0... 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_14_[90882]: created object 25... 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_14_[90882]: created object 49... 2024-01-27T03:49:19.329 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_14_[90882]: finishing. 2024-01-27T03:49:19.330 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_14_[90882]: shutting down. 2024-01-27T03:49:19.330 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.330 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.330 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.330 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:19.330 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: starting. 2024-01-27T03:49:19.330 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: listing objects. 2024-01-27T03:49:19.330 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: listed object 0... 2024-01-27T03:49:24.544 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:10.499 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:24.601 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:10.557 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:24.616 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:10.561 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:24.620 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:10.566 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:24.706 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:10.661 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:24.730 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:10.686 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:24.741 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:10.697 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:24.749 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:10.695 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:24.834 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:10.790 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:25.722 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: listed object 20x55c6df102540 2024-01-27T03:49:25.722 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca3820 2024-01-27T03:49:25.722 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1016e0 2024-01-27T03:49:25.722 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df107180 2024-01-27T03:49:25.722 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df324f60 2024-01-27T03:49:25.722 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df107bf0 2024-01-27T03:49:25.722 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df1080e0 2024-01-27T03:49:25.722 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df106610 2024-01-27T03:49:25.722 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df103420 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df100790 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0fc710 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca6e50 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6dfca7980 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: waiting 0x55c6df0ff2c0 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: done. 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (83126 ms) 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (83126 ms total) 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-01-27T03:49:25.723 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMisc.Exec (95196 ms) 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (8587 ms) 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3457 ms) 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5251 ms) 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (73665 ms) 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] 8 tests from LibRadosMisc (186156 ms total) 2024-01-27T03:49:25.724 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: 2024-01-27T03:49:25.725 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [----------] Global test environment tear-down 2024-01-27T03:49:25.725 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (276309 ms total) 2024-01-27T03:49:25.725 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc: [ PASSED ] 12 tests. 2024-01-27T03:49:25.820 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_1_[41050]: starting. 2024-01-27T03:49:25.820 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_1_[41050]: creating pool ceph_test_rados_delete_pools_parallel.smithi110-40994 2024-01-27T03:49:25.820 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_1_[41050]: created object 0... 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_1_[41050]: created object 25... 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_1_[41050]: created object 49... 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_1_[41050]: finishing. 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_1_[41050]: shutting down. 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_2_[41054]: starting. 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_2_[41054]: deleting pool ceph_test_rados_delete_pools_parallel.smithi110-40994 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_2_[41054]: shutting down. 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: ******************************* 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_3_[63559]: starting. 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_3_[63559]: creating pool ceph_test_rados_delete_pools_parallel.smithi110-40994 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_3_[63559]: created object 0... 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_3_[63559]: created object 25... 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_3_[63559]: created object 49... 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_3_[63559]: finishing. 2024-01-27T03:49:25.821 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_3_[63559]: shutting down. 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: ******************************* 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_5_[63562]: starting. 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_5_[63562]: listing objects. 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_5_[63562]: listed object 0... 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_5_[63562]: listed object 25... 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_5_[63562]: saw 50 objects 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_5_[63562]: shutting down. 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: ******************************* 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_4_[63560]: starting. 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_4_[63560]: deleting pool ceph_test_rados_delete_pools_parallel.smithi110-40994 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: process_4_[63560]: shutting down. 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: ******************************* 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: ******************************* 2024-01-27T03:49:25.822 INFO:tasks.workunit.client.0.smithi110.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-01-27T03:49:25.856 INFO:tasks.workunit.client.0.smithi110.stdout:5... 2024-01-27T03:49:25.856 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: listed object 50... 2024-01-27T03:49:25.856 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: listed object 75... 2024-01-27T03:49:25.856 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: listed object 100... 2024-01-27T03:49:25.856 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: listed object 125... 2024-01-27T03:49:25.856 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: saw 150 objects 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_15_[90884]: shutting down. 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_16_[90885]: starting. 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_16_[90885]: added 25 objects... 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_16_[90885]: added half of the objects 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_16_[90885]: added 50 objects... 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_16_[90885]: added 50 objects 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: process_16_[90885]: shutting down. 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:25.857 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:25.858 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******************************* 2024-01-27T03:49:25.858 INFO:tasks.workunit.client.0.smithi110.stdout: list_parallel: ******* SUCCESS ********** 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: Running main() from gmock_main.cc 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [----------] Global test environment set-up. 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (49071 ms) 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-01-27T03:49:26.979 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (39506 ms) 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (17109 ms) 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.Checksum (20182 ms) 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (48675 ms) 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (14079 ms) 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5540 ms) 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (827 ms) 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.TruncTest (3838 ms) 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-01-27T03:49:26.980 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4300 ms) 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (1713 ms) 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.RmXattr (690 ms) 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIo.XattrIter (560 ms) 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [----------] 14 tests from LibRadosIo (206090 ms total) 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (23040 ms) 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (465 ms) 2024-01-27T03:49:26.981 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (1687 ms) 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (692 ms) 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (492 ms) 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (354 ms) 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (210 ms) 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (243 ms) 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (563 ms) 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (2376 ms) 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [----------] 10 tests from LibRadosIoEC (30122 ms total) 2024-01-27T03:49:26.982 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: 2024-01-27T03:49:26.983 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [----------] Global test environment tear-down 2024-01-27T03:49:26.983 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [==========] 24 tests from 2 test suites ran. (277649 ms total) 2024-01-27T03:49:26.983 INFO:tasks.workunit.client.0.smithi110.stdout: api_io: [ PASSED ] 24 tests. 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: seed 40530 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (28546 ms) 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (51554 ms) 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (11 ms) 2024-01-27T03:49:28.177 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (14866 ms) 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (103970 ms) 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (198947 ms total) 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (7492 ms) 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (2445 ms) 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2494 ms) 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (382 ms) 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (12813 ms total) 2024-01-27T03:49:28.178 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: 2024-01-27T03:49:28.179 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-01-27T03:49:28.179 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (278670 ms total) 2024-01-27T03:49:28.179 INFO:tasks.workunit.client.0.smithi110.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: Running main() from gmock_main.cc 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [----------] Global test environment set-up. 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (18671 ms) 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLock.LockShared (28983 ms) 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (11689 ms) 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (5369 ms) 2024-01-27T03:49:28.180 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (71170 ms) 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLock.Unlock (64636 ms) 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (485 ms) 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (106 ms) 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [----------] 8 tests from LibRadosLock (201109 ms total) 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (8199 ms) 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (464 ms) 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-01-27T03:49:28.181 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (2327 ms) 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1228 ms) 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (157 ms) 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (219 ms) 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (248 ms) 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (165 ms) 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (13008 ms total) 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [----------] Global test environment tear-down 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (278810 ms total) 2024-01-27T03:49:28.182 INFO:tasks.workunit.client.0.smithi110.stdout: api_lock: [ PASSED ] 16 tests. 2024-01-27T03:49:35.108 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify_test_cb 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (39984 ms) 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94384949745888 err -107waiting up to 300 for disconnect notification ... 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: 2024-01-27T03:49:35.109 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (4099 ms) 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94384949745888 err -107 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (22189 ms) 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_cb from 4500 notify_id 377957122049 cookie 94384949745888 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (23471 ms) 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_cb from 4500 notify_id 472446402562 cookie 94384949745888 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (42581 ms) 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_cb from 4500 notify_id 798863917059 cookie 94384949745888 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_cb from 4500 notify_id 837518622724 cookie 94384949745888 2024-01-27T03:49:35.110 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (68708 ms) 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_cb from 4500 notify_id 837518622721 cookie 94384949085696 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_cb from 4500 notify_id 837518622721 cookie 94384949680576 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (1301 ms) 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_cb from 4500 notify_id 841813590021 cookie 94384949767968 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_cb from 4500 notify_id 841813590022 cookie 94384949767968 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (4025 ms) 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94384949767968 err -107 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_cb from 4500 notify_id 850403524610 cookie 94384949767968 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5433 ms) 2024-01-27T03:49:35.111 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94384949765616 err -107 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (2956 ms) 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (214747 ms total) 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: watch_notify_test_cb 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (8488 ms) 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (8488 ms total) 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: 2024-01-27T03:49:35.112 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-01-27T03:49:35.113 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (285565 ms total) 2024-01-27T03:49:35.113 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-01-27T03:49:38.388 INFO:tasks.workunit.client.0.smithi110.stdout: list: Running main() from gmock_main.cc 2024-01-27T03:49:38.389 INFO:tasks.workunit.client.0.smithi110.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-01-27T03:49:38.389 INFO:tasks.workunit.client.0.smithi110.stdout: list: [----------] Global test environment set-up. 2024-01-27T03:49:38.389 INFO:tasks.workunit.client.0.smithi110.stdout: list: [----------] 3 tests from NeoradosList 2024-01-27T03:49:38.389 INFO:tasks.workunit.client.0.smithi110.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-01-27T03:49:38.390 INFO:tasks.workunit.client.0.smithi110.stdout: list: [ OK ] NeoradosList.ListObjects (98272 ms) 2024-01-27T03:49:38.390 INFO:tasks.workunit.client.0.smithi110.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-01-27T03:49:38.390 INFO:tasks.workunit.client.0.smithi110.stdout: list: [ OK ] NeoradosList.ListObjectsNS (119034 ms) 2024-01-27T03:49:38.390 INFO:tasks.workunit.client.0.smithi110.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-01-27T03:49:38.390 INFO:tasks.workunit.client.0.smithi110.stdout: list: [ OK ] NeoradosList.ListObjectsMany (71286 ms) 2024-01-27T03:49:38.391 INFO:tasks.workunit.client.0.smithi110.stdout: list: [----------] 3 tests from NeoradosList (288592 ms total) 2024-01-27T03:49:38.391 INFO:tasks.workunit.client.0.smithi110.stdout: list: 2024-01-27T03:49:38.391 INFO:tasks.workunit.client.0.smithi110.stdout: list: [----------] Global test environment tear-down 2024-01-27T03:49:38.391 INFO:tasks.workunit.client.0.smithi110.stdout: list: [==========] 3 tests from 1 test suite ran. (288592 ms total) 2024-01-27T03:49:38.391 INFO:tasks.workunit.client.0.smithi110.stdout: list: [ PASSED ] 3 tests. 2024-01-27T03:49:43.974 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:29.929 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:43.977 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:29.933 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:44.030 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:29.975 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:44.032 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:29.977 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:44.038 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:29.994 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:44.048 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:30.004 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:44.068 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:30.024 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:44.118 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:30.064 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:44.144 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:30.100 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: Running main() from gmock_main.cc 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [----------] Global test environment set-up. 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: seed 40187 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (18957 ms) 2024-01-27T03:49:46.066 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (83746 ms) 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (2272 ms) 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (2126 ms) 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (13887 ms) 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (46026 ms) 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3573 ms) 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (32097 ms) 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-01-27T03:49:46.067 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (2161 ms) 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (249 ms) 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (190 ms) 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (269 ms) 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (378 ms) 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (443 ms) 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (2460 ms) 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-01-27T03:49:46.068 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (133 ms) 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (93 ms) 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (833 ms) 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (58 ms) 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (209952 ms total) 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (15010 ms) 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (8414 ms) 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-01-27T03:49:46.069 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (2361 ms) 2024-01-27T03:49:46.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-01-27T03:49:46.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (272 ms) 2024-01-27T03:49:46.070 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-01-27T03:49:59.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.R api_watch_notify_pp: Running main() from gmock_main.cc 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: notify 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (98715 ms) 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (101225 ms) 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (199940 ms total) 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: notify 2024-01-27T03:49:59.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (530 ms) 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: notify 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (30261 ms) 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (2876 ms) 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (2074 ms) 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194955559216 notify_id 970662608897 notifier_gid 10168 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (318 ms) 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194955559216 notify_id 979252543490 notifier_gid 10168 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (2383 ms) 2024-01-27T03:49:59.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194955559216 notify_id 987842478083 notifier_gid 10168 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (2807 ms) 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194955559216 notify_id 996432412675 notifier_gid 10168 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (1574 ms) 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194955559216 notify_id 1005022347268 notifier_gid 10168 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (2885 ms) 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194955559216 notify_id 1026497183748 notifier_gid 10168 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (4353 ms) 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: trying... 2024-01-27T03:49:59.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194954746896 notify_id 1035087118341 notifier_gid 10168 2024-01-27T03:49:59.041 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: timed out 2024-01-27T03:50:04.372 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:50.328 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:04.379 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:50.335 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:04.403 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:50.359 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:04.443 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:50.388 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:04.443 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:50.400 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:04.457 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:50.403 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:04.507 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:50.462 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:04.548 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:50.504 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:04.552 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:50.498 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:06.289 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:52.245 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:06.294 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:52.250 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:06.307 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:52.253 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:06.309 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:52.254 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:06.335 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:07:52.281 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:06.381 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:07:52.337 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:07.046 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:53.002 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:07.056 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:53.012 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:07.086 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:07:53.042 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:09.033 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_ api_snapshots: Running main() from gmock_main.cc 2024-01-27T03:50:09.033 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-01-27T03:50:09.033 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] Global test environment set-up. 2024-01-27T03:50:09.033 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-01-27T03:50:09.033 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (22634 ms) 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (44040 ms) 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (10716 ms) 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (24053 ms) 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (101443 ms total) 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (34910 ms) 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (24784 ms) 2024-01-27T03:50:09.034 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (13969 ms) 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (73663 ms total) 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (31766 ms) 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (28481 ms) 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (25165 ms) 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-01-27T03:50:09.035 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (4540 ms) 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (89952 ms total) 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (15769 ms) 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (14363 ms) 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (30132 ms total) 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [----------] Global test environment tear-down 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (319561 ms total) 2024-01-27T03:50:09.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-01-27T03:50:14.791 INFO:tasks.workunit.client.0.smithi110.stdout: io: Running main() from gmock_main.cc 2024-01-27T03:50:14.791 INFO:tasks.workunit.client.0.smithi110.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-01-27T03:50:14.791 INFO:tasks.workunit.client.0.smithi110.stdout: io: [----------] Global test environment set-up. 2024-01-27T03:50:14.791 INFO:tasks.workunit.client.0.smithi110.stdout: io: [----------] 14 tests from NeoRadosIo 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.Limits (51925 ms) 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (28854 ms) 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.ReadOp (83952 ms) 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.SparseRead (53223 ms) 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.RoundTrip (30269 ms) 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (27786 ms) 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-01-27T03:50:14.792 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (6555 ms) 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (8332 ms) 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (5945 ms) 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.Trunc (6809 ms) 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.Remove (5048 ms) 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (7134 ms) 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-01-27T03:50:14.793 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.RmXattr (5053 ms) 2024-01-27T03:50:14.794 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-01-27T03:50:14.794 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ OK ] NeoRadosIo.GetXattrs (4138 ms) 2024-01-27T03:50:14.794 INFO:tasks.workunit.client.0.smithi110.stdout: io: [----------] 14 tests from NeoRadosIo (325023 ms total) 2024-01-27T03:50:14.794 INFO:tasks.workunit.client.0.smithi110.stdout: io: 2024-01-27T03:50:14.794 INFO:tasks.workunit.client.0.smithi110.stdout: io: [----------] Global test environment tear-down 2024-01-27T03:50:14.794 INFO:tasks.workunit.client.0.smithi110.stdout: io: [==========] 14 tests from 1 test suite ran. (325024 ms total) 2024-01-27T03:50:14.794 INFO:tasks.workunit.client.0.smithi110.stdout: io: [ PASSED ] 14 tests. 2024-01-27T03:50:21.768 INFO:tasks.workunit.client.0.smithi110.stdout:notify_pp: flushing 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: flushed 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (4293 ms) 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: trying... 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194954746896 notify_id 1060856922118 notifier_gid 10168 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: timed out 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: flushing 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: flushed 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (5688 ms) 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: List watches 2024-01-27T03:50:21.769 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: notify2 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194954746896 notify_id 1082331758599 notifier_gid 10168 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: notify2 done 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: watch_check 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: unwatch2 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: flushing 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: done 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (7509 ms) 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: List watches 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: notify2 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: handle_notify cookie 94194954746896 notify_id 1103806595080 notifier_gid 10168 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: notify2 done 2024-01-27T03:50:21.770 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: watch_check 2024-01-27T03:50:21.771 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: unwatch2 2024-01-27T03:50:21.771 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: flushing 2024-01-27T03:50:21.771 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: done 2024-01-27T03:50:21.771 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (4510 ms) 2024-01-27T03:50:21.771 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (72061 ms total) 2024-01-27T03:50:21.771 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: 2024-01-27T03:50:21.771 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-01-27T03:50:21.771 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (332244 ms total) 2024-01-27T03:50:21.771 INFO:tasks.workunit.client.0.smithi110.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-01-27T03:50:24.625 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:10.581 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:24.663 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:10.618 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:24.663 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:08:10.607 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:24.673 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:08:10.619 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:24.700 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:10.655 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:24.708 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:10.664 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:24.762 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:08:10.708 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:24.765 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:10.721 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout:oundTripPP2 (684 ms) 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (346 ms) 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (3056 ms) 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2804 ms) 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (3849 ms) 2024-01-27T03:50:24.799 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (2075 ms) 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (5476 ms) 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3625 ms) 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9429 ms) 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (850 ms) 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (2267 ms) 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (1417 ms) 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-01-27T03:50:24.800 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (224 ms) 2024-01-27T03:50:24.801 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (62159 ms total) 2024-01-27T03:50:24.801 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: 2024-01-27T03:50:24.801 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [----------] Global test environment tear-down 2024-01-27T03:50:24.801 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (335460 ms total) 2024-01-27T03:50:24.801 INFO:tasks.workunit.client.0.smithi110.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-01-27T03:50:24.808 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:10.764 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:28.029 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: Running main() from gmock_main.cc 2024-01-27T03:50:28.029 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-01-27T03:50:28.029 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [----------] Global test environment set-up. 2024-01-27T03:50:28.029 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (36030 ms) 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (62334 ms) 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (59441 ms) 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (10039 ms) 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (25232 ms) 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (193076 ms total) 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (44901 ms) 2024-01-27T03:50:28.030 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (46242 ms) 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (15077 ms) 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (10103 ms) 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (7432 ms) 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi110-41333-11. 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: Waiting for snaps to purge. 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (21358 ms) 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (145114 ms total) 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: 2024-01-27T03:50:28.031 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [----------] Global test environment tear-down 2024-01-27T03:50:28.032 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (338190 ms total) 2024-01-27T03:50:28.032 INFO:tasks.workunit.client.0.smithi110.stdout: snapshots: [ PASSED ] 11 tests. 2024-01-27T03:50:36.264 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-01-27T03:50:36.264 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-01-27T03:50:36.264 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-01-27T03:50:36.264 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-01-27T03:50:36.264 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: seed 40481 2024-01-27T03:50:36.264 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (15337 ms) 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (16103 ms) 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (39212 ms) 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (9493 ms) 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (57301 ms) 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (137446 ms total) 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (88050 ms) 2024-01-27T03:50:36.265 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (42486 ms) 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (19935 ms) 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (6632 ms) 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (5008 ms) 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-926-g5ffe0641/rpm/el9/BUILD/ceph-19.0.0-926-g5ffe0641/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi110-40481-7 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: waiting for snaps to purge 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (23645 ms) 2024-01-27T03:50:36.266 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (185756 ms total) 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (6 ms) 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (6443 ms) 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (6449 ms total) 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3443 ms) 2024-01-27T03:50:36.267 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-01-27T03:50:39.475 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] Li ec_io: Running main() from gmock_main.cc 2024-01-27T03:50:39.475 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-01-27T03:50:39.475 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [----------] Global test environment set-up. 2024-01-27T03:50:39.475 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-01-27T03:50:39.475 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-01-27T03:50:39.475 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (10361 ms) 2024-01-27T03:50:39.475 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-01-27T03:50:39.475 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (339327 ms) 2024-01-27T03:50:39.475 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (349688 ms total) 2024-01-27T03:50:39.476 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: 2024-01-27T03:50:39.476 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [----------] Global test environment tear-down 2024-01-27T03:50:39.476 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (349688 ms total) 2024-01-27T03:50:39.476 INFO:tasks.workunit.client.0.smithi110.stdout: ec_io: [ PASSED ] 2 tests. 2024-01-27T03:50:39.489 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: Running main() from gmock_main.cc 2024-01-27T03:50:39.489 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-01-27T03:50:39.489 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [----------] Global test environment set-up. 2024-01-27T03:50:39.489 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-01-27T03:50:39.489 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-01-27T03:50:39.489 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (74266 ms) 2024-01-27T03:50:39.489 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-01-27T03:50:39.489 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (21731 ms) 2024-01-27T03:50:39.489 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (45060 ms) 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (76500 ms) 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (41359 ms) 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (29666 ms) 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (10485 ms) 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (12525 ms) 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-01-27T03:50:39.490 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (6503 ms) 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (5828 ms) 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (9086 ms) 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (7865 ms) 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (4497 ms) 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (4292 ms) 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (349663 ms total) 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [----------] Global test environment tear-down 2024-01-27T03:50:39.491 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (349663 ms total) 2024-01-27T03:50:39.492 INFO:tasks.workunit.client.0.smithi110.stdout: read_operations: [ PASSED ] 14 tests. 2024-01-27T03:50:44.741 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:30.697 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:44.743 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:30.699 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:44.799 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:30.755 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:44.802 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:30.758 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:44.808 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:30.764 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:44.816 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:08:30.762 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:44.819 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:08:30.764 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:44.867 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:30.823 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:50:44.890 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:08:30.836 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:01.224 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: Running main() from gmock_main.cc 2024-01-27T03:51:01.224 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-01-27T03:51:01.224 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [----------] Global test environment set-up. 2024-01-27T03:51:01.224 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.TooBig (7151 ms) 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (51907 ms) 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (36413 ms) 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (44996 ms) 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (9630 ms) 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-01-27T03:51:01.225 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (39649 ms) 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (8439 ms) 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (19856 ms) 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (41579 ms) 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (33023 ms) 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (4653 ms) 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-01-27T03:51:01.226 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (6807 ms) 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (5006 ms) 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (5808 ms) 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.Flush (6381 ms) 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (6270 ms) 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4872 ms) 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (6266 ms) 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-01-27T03:51:01.227 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (4915 ms) 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (4352 ms) 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3698 ms) 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4488 ms) 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3345 ms) 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4505 ms) 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3576 ms) 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-01-27T03:51:01.228 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4314 ms) 2024-01-27T03:51:01.229 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [----------] 26 tests from LibRadosAio (371901 ms total) 2024-01-27T03:51:01.229 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: 2024-01-27T03:51:01.229 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-01-27T03:51:05.062 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:51.017 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:05.067 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:51.023 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:05.120 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:08:51.066 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:05.123 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:51.079 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:05.126 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:51.083 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:05.136 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:08:51.082 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:05.192 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:08:51.137 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:05.193 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:08:51.150 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:05.215 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:08:51.171 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:07.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.SimplbRadosSnapshotsECPP.SnapRemovePP (2098 ms) 2024-01-27T03:51:07.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-01-27T03:51:07.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (3253 ms) 2024-01-27T03:51:07.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-01-27T03:51:07.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (3577 ms) 2024-01-27T03:51:07.013 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (12371 ms total) 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (5459 ms) 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4526 ms) 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4139 ms) 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (14125 ms total) 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (377548 ms total) 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-01-27T03:51:07.014 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-01-27T03:51:07.015 INFO:tasks.workunit.client.0.smithi110.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-01-27T03:51:11.146 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: Running main() from gmock_main.cc 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [----------] Global test environment set-up. 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (30285 ms) 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (39772 ms) 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (20631 ms) 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (19465 ms) 2024-01-27T03:51:11.147 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (36428 ms) 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (13278 ms) 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (221428 ms) 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (381288 ms total) 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [----------] Global test environment tear-down 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (381288 ms total) 2024-01-27T03:51:11.148 INFO:tasks.workunit.client.0.smithi110.stdout: write_operations: [ PASSED ] 7 tests. 2024-01-27T03:51:14.834 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=1 expected=1 2024-01-27T03:51:14.834 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:de5d7c5f:::12:head 2024-01-27T03:51:14.834 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:de5d7c5f:::12:head expected=16:de5d7c5f:::12:head 2024-01-27T03:51:14.834 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:de5d7c5f:::12:head -> 12 2024-01-27T03:51:14.834 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=12 expected=12 2024-01-27T03:51:14.834 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:d83876eb:::4:head 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:d83876eb:::4:head expected=16:d83876eb:::4:head 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:d83876eb:::4:head -> 4 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=4 expected=4 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:cfc208b3:::3:head 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:cfc208b3:::3:head expected=16:cfc208b3:::3:head 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:cfc208b3:::3:head -> 3 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=3 expected=3 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:c4fdafeb:::6:head 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:c4fdafeb:::6:head expected=16:c4fdafeb:::6:head 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:c4fdafeb:::6:head -> 6 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=6 expected=6 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:bd63b0f1:::8:head 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:bd63b0f1:::8:head expected=16:bd63b0f1:::8:head 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:bd63b0f1:::8:head -> 8 2024-01-27T03:51:14.835 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=8 expected=8 2024-01-27T03:51:14.836 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:6cac518f:::0:head 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:6cac518f:::0:head expected=16:6cac518f:::0:head 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:6cac518f:::0:head -> 0 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=0 expected=0 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:f905c69b:::2:head 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:f905c69b:::2:head expected=16:f905c69b:::2:head 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:f905c69b:::2:head -> 2 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=2 expected=2 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : seek to 16:e960b815:::9:head 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : cursor()=16:e960b815:::9:head expected=16:e960b815:::9:head 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: > 16:e960b815:::9:head -> 9 2024-01-27T03:51:14.839 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: : entry=9 expected=9 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (13084 ms) 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (115544 ms) 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: split 0/5 -> MIN 16:33333333::::head 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: split 1/5 -> 16:33333333::::head 16:66666666::::head 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: split 2/5 -> 16:66666666::::head 16:99999999::::head 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: split 3/5 -> 16:99999999::::head 16:cccccccc::::head 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: split 4/5 -> 16:cccccccc::::head MAX 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (16191 ms) 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [----------] 7 tests from LibRadosList (368380 ms total) 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 2024-01-27T03:51:14.840 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1901 ms) 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: myset foo1,foo2,foo3 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo1 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo2 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo3 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: myset foo1,foo4,foo5 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo4 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo5 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo1 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: myset foo6,foo7 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo7 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: foo6 2024-01-27T03:51:14.841 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-01-27T03:51:14.842 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns1:foo4 2024-01-27T03:51:14.842 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns1:foo5 2024-01-27T03:51:14.842 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns2:foo7 2024-01-27T03:51:14.842 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns2:foo6 2024-01-27T03:51:14.842 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: ns1:foo1 2024-01-27T03:51:14.842 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: :foo1 2024-01-27T03:51:14.842 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: :foo2 2024-01-27T03:51:14.842 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: :foo3 2024-01-27T03:51:14.842 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (3855 ms) 2024-01-27T03:51:24.889 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:10.845 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:24.949 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:10.905 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:24.956 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:10.912 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:24.963 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:10.919 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:25.010 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:10.966 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:25.033 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:10.989 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:25.045 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:10.990 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:25.049 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:10.995 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:25.096 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:11.042 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:25.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-01-27T03:51:25.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 1 0 2024-01-27T03:51:25.662 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 10 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 13 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 7 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 14 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 0 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 15 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 11 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 5 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 8 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 6 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 3 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 4 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 12 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 9 0 2024-01-27T03:51:25.663 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 2 0 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (3604 ms) 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [----------] 3 tests from LibRadosListEC (9360 ms total) 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3630 ms) 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [----------] 1 test from LibRadosListNP (3630 ms total) 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [----------] Global test environment tear-down 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [==========] 11 tests from 3 test suites ran. (396271 ms total) 2024-01-27T03:51:25.664 INFO:tasks.workunit.client.0.smithi110.stdout: api_list: [ PASSED ] 11 tests. 2024-01-27T03:51:44.996 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:30.941 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:44.996 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:30.952 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:44.997 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:30.942 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:45.001 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:30.958 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:45.046 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:30.992 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:45.059 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:31.015 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:45.063 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:31.019 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:45.065 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:31.021 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:45.119 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:31.074 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:52.901 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:38.857 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:52.954 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:38.909 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:52.964 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:38.920 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:52.967 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:38.923 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:52.969 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:38.915 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:52.973 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:38.919 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:52.987 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:38.943 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:52.999 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:38.955 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:51:53.005 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:38.950 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.072 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:51.028 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.129 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:51.084 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.173 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:51.118 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.174 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:09:51.130 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.176 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:51.121 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.186 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:51.142 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.191 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:51.147 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.206 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:09:51.152 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.224 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:09:51.180 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:05.839 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: Running main() from gmock_main.cc 2024-01-27T03:52:05.839 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-01-27T03:52:05.839 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [----------] Global test environment set-up. 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (100704 ms) 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (133858 ms) 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (201470 ms) 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [----------] 3 tests from NeoradosECList (436032 ms total) 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [----------] Global test environment tear-down 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (436032 ms total) 2024-01-27T03:52:05.840 INFO:tasks.workunit.client.0.smithi110.stdout: ec_list: [ PASSED ] 3 tests. 2024-01-27T03:52:17.287 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-01-27T03:52:17.287 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-01-27T03:52:17.287 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-01-27T03:52:17.287 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-01-27T03:52:17.287 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-01-27T03:52:17.287 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (28192 ms) 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (47449 ms) 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (44151 ms) 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (97995 ms) 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (41361 ms) 2024-01-27T03:52:17.288 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-01-27T03:52:17.289 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (22286 ms) 2024-01-27T03:52:17.289 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-01-27T03:52:17.289 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (166245 ms) 2024-01-27T03:52:17.289 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (447680 ms total) 2024-01-27T03:52:17.289 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: 2024-01-27T03:52:17.289 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-01-27T03:52:17.289 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (447680 ms total) 2024-01-27T03:52:17.289 INFO:tasks.workunit.client.0.smithi110.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-01-27T03:52:25.362 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:11.317 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:25.438 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:11.394 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:25.498 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:11.454 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:25.541 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:10:11.486 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:25.542 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:10:11.488 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:25.551 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:11.508 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:25.558 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:11.515 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:25.593 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:10:11.539 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:25.619 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:11.575 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:45.437 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:31.393 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:45.440 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:31.396 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:45.485 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:31.441 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:45.488 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:31.445 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:45.489 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:31.446 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:45.516 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:10:31.461 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:45.518 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:10:31.463 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:45.546 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:31.502 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:45.591 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:10:31.536 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:52:59.748 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-01-27T03:52:59.748 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-01-27T03:52:59.748 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-01-27T03:52:59.748 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-01-27T03:52:59.748 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-01-27T03:52:59.748 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-01-27T03:52:59.748 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: 2024-01-27T03:52:59.748 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-01-27T03:52:59.748 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: seed 40364 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (324 ms) 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (117297 ms) 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (18363 ms) 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (32904 ms) 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (16242 ms) 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2421 ms) 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-01-27T03:52:59.749 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3676 ms) 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (3033 ms) 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (5668 ms) 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (427 ms) 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (690 ms) 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (1850 ms) 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (425 ms) 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-01-27T03:52:59.750 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: osd_max_attr_size = 0 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (195470 ms) 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (8163 ms) 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: done waiting, doing copies 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: waiting for final deep scrubs... 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: done waiting 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (72936 ms) 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (82 ms) 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (41 ms) 2024-01-27T03:52:59.751 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-01-27T03:52:59.752 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4345 ms) 2024-01-27T03:52:59.752 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-01-27T03:52:59.752 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-01-27T03:52:59.752 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-01-27T03:52:59.752 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-01-27T03:52:59.752 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-01-27T03:52:59.752 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-01-27T03:53:05.552 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [-eWrite 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7853 ms) 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7224 ms) 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (8051 ms) 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (8020 ms) 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7219 ms) 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (9129 ms) 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-01-27T03:53:05.553 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7751 ms) 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7030 ms) 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7985 ms) 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (8056 ms) 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7048 ms) 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (8178 ms) 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (8058 ms) 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7871 ms) 2024-01-27T03:53:05.554 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-01-27T03:53:05.555 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (8050 ms) 2024-01-27T03:53:05.555 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-01-27T03:53:05.555 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6803 ms) 2024-01-27T03:53:05.555 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (124326 ms total) 2024-01-27T03:53:05.555 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: 2024-01-27T03:53:05.555 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [----------] Global test environment tear-down 2024-01-27T03:53:05.555 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (496228 ms total) 2024-01-27T03:53:05.555 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio: [ PASSED ] 42 tests. 2024-01-27T03:53:05.636 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:51.592 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:05.640 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:51.596 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:05.687 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:10:51.643 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:05.691 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:51.647 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:05.695 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:51.651 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:05.700 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:10:51.646 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:05.701 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:10:51.647 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:05.740 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:10:51.696 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:05.769 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:10:51.715 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:25.631 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:11.587 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:25.636 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:11.592 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:25.688 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:11.644 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:25.707 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:11.653 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:25.708 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:11.654 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:25.711 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:11.667 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:25.714 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:11.670 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:25.750 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:11.696 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:25.775 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:11.731 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:33.232 INFO:tasks.workunit.client.0.smithi110.stdout:---------] 22 tests from LibRadosMiscPP (484357 ms total) 2024-01-27T03:53:33.232 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: 2024-01-27T03:53:33.232 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-01-27T03:53:33.232 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-01-27T03:53:33.232 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (5785 ms) 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (5785 ms total) 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (591 ms) 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (53 ms) 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (644 ms total) 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (853 ms) 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-01-27T03:53:33.233 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (47 ms) 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (900 ms total) 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (664 ms) 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (43 ms) 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (707 ms total) 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1514 ms) 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1514 ms total) 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: 2024-01-27T03:53:33.234 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-01-27T03:53:33.235 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (523786 ms total) 2024-01-27T03:53:33.235 INFO:tasks.workunit.client.0.smithi110.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-01-27T03:53:43.149 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:29.105 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:43.150 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:29.106 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:43.158 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:29.114 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:43.192 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:29.148 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:43.198 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:29.144 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:43.203 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:29.148 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:43.214 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:29.170 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:43.250 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:29.195 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:43.273 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:29.230 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:45.616 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:31.572 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:45.620 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:31.576 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:45.633 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:31.590 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:45.671 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:31.617 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:45.671 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:31.628 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:45.674 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:31.620 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:45.675 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:31.631 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:45.686 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:31.632 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:53:45.688 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:31.644 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:05.697 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:51.653 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:05.699 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:51.656 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:05.735 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:11:51.691 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:05.746 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:51.692 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:05.747 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:51.702 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:05.751 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:51.707 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:05.752 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:51.697 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:05.788 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:11:51.733 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:05.790 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:11:51.746 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:25.830 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:11.786 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:25.832 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:11.789 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:25.883 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:11.839 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:25.884 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:11.840 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:25.886 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:11.843 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:25.887 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:11.833 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:25.890 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:11.835 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:25.926 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:11.882 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:25.927 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:11.873 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:31.037 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:16.992 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:31.043 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:16.999 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:31.063 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:17.019 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:31.092 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:17.038 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:31.094 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:17.039 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:31.111 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:17.068 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:31.113 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:17.070 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:31.115 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:17.061 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:31.139 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:17.095 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:40.962 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:26.918 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:41.026 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:26.982 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:41.027 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:26.973 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:41.038 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:26.984 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:41.065 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:27.010 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:41.073 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:27.030 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:41.089 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:27.045 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:41.090 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:27.046 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:41.118 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:27.074 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:50.960 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:36.916 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:50.962 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:36.919 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:50.993 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:36.949 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:51.010 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:36.956 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:51.017 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:36.962 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:51.033 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:36.989 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:51.037 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:36.993 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:51.039 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:36.985 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:51.057 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:37.013 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:55.916 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:41.873 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:55.963 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:41.909 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:55.967 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:41.923 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:55.969 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:41.915 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:55.983 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:41.928 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:55.984 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:41.940 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:56.016 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:41.973 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:56.020 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:41.977 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:54:56.033 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:41.989 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:11.157 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:57.113 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:11.162 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:57.119 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:11.197 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:12:57.153 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:11.222 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:57.167 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:11.232 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:57.177 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:11.232 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:57.189 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:11.234 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:57.190 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:11.252 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:12:57.208 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:11.254 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:12:57.200 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:16.143 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:02.099 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:16.198 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:02.154 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:16.202 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:02.158 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:16.204 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:02.150 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:16.206 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:02.163 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:16.214 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:02.160 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:16.220 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:02.176 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:16.225 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:02.182 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:16.238 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:02.183 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:31.118 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:17.073 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:31.122 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:17.078 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:31.153 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:17.109 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:31.168 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:17.124 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:31.170 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:17.127 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:31.183 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:17.129 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:31.186 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:17.132 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:31.215 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:17.170 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:31.217 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:17.163 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:32.152 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:18.108 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:32.153 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:18.110 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:32.164 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:18.121 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:32.198 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:18.143 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:32.199 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:18.144 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:32.209 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:18.154 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:32.240 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:18.196 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:32.241 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:18.198 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:32.258 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:18.214 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:46.167 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:32.123 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:46.170 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:32.126 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:46.203 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:32.159 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:46.217 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:32.173 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:46.220 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:32.166 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:46.221 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:32.177 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:46.225 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:32.171 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:46.251 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:32.207 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:46.256 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:32.202 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:51.404 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:37.360 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:51.408 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:37.364 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:51.437 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:37.393 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:51.457 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:37.403 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:51.459 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:37.405 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:51.466 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:37.422 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:51.467 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:37.423 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:51.478 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:37.424 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:55:51.487 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:37.443 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:01.284 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:47.240 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:01.286 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:47.242 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:01.311 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:47.268 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:01.333 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:47.290 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:01.334 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:47.279 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:01.336 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:47.293 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:01.354 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:47.300 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:01.367 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:47.323 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:01.379 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:47.325 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:11.363 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:57.319 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:11.414 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:57.370 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:11.440 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:13:57.396 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:11.473 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:57.418 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:11.474 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:57.431 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:11.476 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:57.421 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:11.480 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:57.436 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:11.499 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:13:57.445 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:11.507 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:13:57.464 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:15.351 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-01-27T03:56:15.351 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-01-27T03:56:15.351 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-01-27T03:56:15.351 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: seed 40469 2024-01-27T03:56:15.351 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-01-27T03:56:15.351 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (52518 ms) 2024-01-27T03:56:15.351 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-01-27T03:56:15.351 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (107805 ms) 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (392 ms) 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (160715 ms total) 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (92330 ms) 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (70344 ms) 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (25979 ms) 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: my_snaps [3] 2024-01-27T03:56:15.352 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: my_snaps [4,3] 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: my_snaps [5,4,3] 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: promoting some heads 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: promoting from clones for snap 6 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: promoting from clones for snap 5 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: promoting from clones for snap 4 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: promoting from clones for snap 3 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: waiting for scrubs... 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: done waiting 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (151180 ms) 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10937 ms) 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8853 ms) 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-01-27T03:56:15.353 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (9241 ms) 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (11127 ms) 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12604 ms) 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9909 ms) 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (13311 ms) 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (39576 ms) 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10591 ms) 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (11966 ms) 2024-01-27T03:56:15.354 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-01-27T03:56:15.355 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (15454 ms) 2024-01-27T03:56:15.355 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-01-27T03:56:15.355 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (10528 ms) 2024-01-27T03:56:15.355 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-01-27T03:56:15.355 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11257 ms) 2024-01-27T03:56:31.398 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:17.354 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:31.401 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:17.357 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:31.448 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:17.393 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:31.449 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:17.395 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:31.456 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:17.412 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:31.465 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:17.422 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:31.467 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:17.423 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:31.488 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:17.434 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:31.504 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:17.460 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:36.520 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:22.475 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:36.576 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:22.532 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:36.592 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:22.548 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:36.616 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:22.561 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:36.621 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:22.567 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:36.635 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:22.581 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:36.639 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:22.595 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:36.642 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:22.598 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:36.660 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:22.616 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:37.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-01-27T03:56:37.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: hmm, no HitSet yet 2024-01-27T03:56:37.932 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: ok, hit_set contains 210:602f83fe:::foo:head 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9516 ms) 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg_num = 32 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 0 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 1 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 2 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 3 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 4 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 5 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 6 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 7 ls 1706327792,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 8 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 9 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 10 ls 1706327793,0 2024-01-27T03:56:37.933 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 11 ls 1706327793,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 12 ls 1706327792,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 13 ls 1706327793,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 14 ls 1706327792,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 15 ls 1706327793,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 16 ls 1706327792,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 17 ls 1706327793,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 18 ls 1706327792,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 19 ls 1706327793,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 20 ls 1706327793,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 21 ls 1706327793,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 22 ls 1706327793,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 23 ls 1706327793,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 24 ls 1706327792,0 2024-01-27T03:56:37.934 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 25 ls 1706327793,0 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 26 ls 1706327792,0 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 27 ls 1706327793,0 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 28 ls 1706327793,0 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 29 ls 1706327793,0 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 30 ls 1706327792,0 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg 31 ls 1706327792,0 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: pg_num = 32 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:6cac518f:::0:head 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:02547ec2:::1:head 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:f905c69b:::2:head 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:cfc208b3:::3:head 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d83876eb:::4:head 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:b29083e3:::5:head 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c4fdafeb:::6:head 2024-01-27T03:56:37.935 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:5c6b0b28:::7:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:bd63b0f1:::8:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:e960b815:::9:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:52ea6a34:::10:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:89d3ae78:::11:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:de5d7c5f:::12:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:566253c9:::13:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:62a1935d:::14:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:863748b0:::15:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:3958e169:::16:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:4d4dabf9:::17:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:8391935d:::18:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:28883081:::19:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:69259c59:::20:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:4bdb80b7:::21:head 2024-01-27T03:56:37.936 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:a11c5d71:::22:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:271af37b:::23:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:95b121be:::24:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:58d1031b:::25:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:0a050783:::26:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c709704c:::27:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:cbe56eaf:::28:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:86b4b162:::29:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:70d89383:::30:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:dd450c7c:::31:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:6d5729b1:::32:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c388f3fb:::33:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:56cfea31:::34:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:9dbc1bf7:::35:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:40b74ccd:::36:head 2024-01-27T03:56:37.937 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:4d5aaf42:::37:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:920f362c:::38:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:6cc53222:::39:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:9cad833f:::40:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:1ea84d41:::41:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c4480ef6:::42:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:a694361e:::43:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d1bd33e9:::44:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:ddc2cd5d:::45:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:2b782207:::46:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:7b187fca:::47:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:90ecdf6f:::48:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:a5ed95fe:::49:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:ea0eaa55:::50:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:f33ef17b:::51:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:a0d1b2f6:::52:head 2024-01-27T03:56:37.938 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:60c5229e:::53:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:edcbc575:::54:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:102cf253:::55:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:efb7fb0b:::56:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:50d0a326:::57:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d4dc5daf:::58:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:3a130462:::59:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:ec87ed71:::60:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d5bc9454:::61:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:3ddfe313:::62:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:7c2816b9:::63:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:47e00e4d:::64:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c6410c18:::65:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:b48ed237:::66:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:cd63ad31:::67:head 2024-01-27T03:56:37.939 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:b179e92b:::68:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:0d9f741a:::69:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:6d3352ae:::70:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c6d5c19e:::71:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:bc4729c3:::72:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:77e930b9:::73:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:0abeecfd:::74:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:b7c37e15:::75:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:b6378398:::76:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:02bd68de:::77:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:cc795d2d:::78:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:630d4fea:::79:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:e0d29ef5:::80:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:fd6f13d2:::81:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:606461d5:::82:head 2024-01-27T03:56:37.940 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:eadbdc43:::83:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:8761d0bb:::84:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:9ef0186f:::85:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:e0d41294:::86:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:961de695:::87:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:1423148f:::88:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:633a8fa2:::89:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:a8653809:::90:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:3dac8b33:::91:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:35aad435:::92:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:f6dcc343:::93:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:dbbdad87:::94:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:1cb48ce0:::95:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:03cd461c:::96:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:17a4ea99:::97:head 2024-01-27T03:56:37.941 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:9993c9a7:::98:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:6394211c:::99:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:94c7ae57:::100:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:6fdee5bb:::101:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:9a477fd1:::102:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:eb850916:::103:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:affc56b9:::104:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:b42dc814:::105:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:f319f8f0:::106:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:9a40b9de:::107:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:8b524f28:::108:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:e3de589f:::109:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:90f90a5b:::110:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:a7b4f1d7:::111:head 2024-01-27T03:56:37.942 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:af51766e:::112:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:b6f90bd1:::113:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:e0261208:::114:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c9569ef7:::115:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:61bebe50:::116:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:fe93412b:::117:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d3d38bee:::118:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:3100ba0c:::119:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d0560ada:::120:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:f0ea8b35:::121:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:766f231a:::122:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:a07a2582:::123:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:bd7c6b3a:::124:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:fb2ddaff:::125:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:4408e1fe:::126:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:ee1df7a7:::127:head 2024-01-27T03:56:37.943 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c3002909:::128:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:4f48ffa9:::129:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:edf38733:::130:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c08425c0:::131:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:5f902d98:::132:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:41ea2c93:::133:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:813cee13:::134:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:0131818d:::135:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:26ba5a85:::136:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:381b8a5a:::137:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:28797e47:::138:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:bfca7f22:::139:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:36807075:::140:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:80b03975:::141:head 2024-01-27T03:56:37.944 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:5c15709b:::142:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:f39ea15e:::143:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:ea992956:::144:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:48887b1c:::145:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:9f24a9dd:::146:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:987f100b:::147:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d2dd3581:::148:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:7fed1808:::149:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c80b70e9:::150:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:85ed90f9:::151:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:36428b24:::152:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d044c34a:::153:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:7c18bf58:::154:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d1c21232:::155:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:a7a3c575:::156:head 2024-01-27T03:56:37.945 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:87da0633:::157:head 2024-01-27T03:56:37.946 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d5ac3822:::158:head 2024-01-27T03:56:37.946 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:3f20522d:::159:head 2024-01-27T03:56:37.946 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:6ca26563:::160:head 2024-01-27T03:56:37.946 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:532ce135:::161:head 2024-01-27T03:56:37.946 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:c78863e6:::162:head 2024-01-27T03:56:51.472 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:37.428 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:51.477 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:37.433 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:51.502 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:37.458 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:51.521 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:37.467 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:51.523 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:37.479 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:51.524 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:37.470 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:51.527 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:37.484 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:51.547 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:37.493 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:51.551 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:37.507 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:52.415 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:38.371 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:52.416 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:38.372 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:52.426 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:38.382 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:52.429 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:38.375 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:52.431 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:38.376 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:52.440 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:38.386 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:52.525 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:38.481 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:52.526 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:38.482 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:56:52.541 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:38.497 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:06.527 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:52.483 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:06.530 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:52.487 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:06.559 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:52.515 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:06.576 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:52.522 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:06.578 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:52.523 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:06.585 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:52.541 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:06.590 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:52.547 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:06.596 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:52.542 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:06.610 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:52.566 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:11.766 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:57.722 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:11.769 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:57.726 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:11.792 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:14:57.748 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:11.827 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:57.783 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:11.831 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:57.787 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:11.844 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:57.789 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:11.850 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:14:57.806 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:11.851 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:57.797 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:11.872 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:14:57.818 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:21.674 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:07.630 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:21.751 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:07.707 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:21.787 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:07.743 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:21.801 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:07.747 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:21.805 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:07.751 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:21.807 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:07.763 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:21.810 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:07.765 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:21.842 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:07.798 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:21.853 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:07.798 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:31.843 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:17.799 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:31.852 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:17.808 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:31.903 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:17.859 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:31.918 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:17.874 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:31.940 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:17.896 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:31.951 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:17.897 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:31.958 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:17.904 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:31.966 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:17.922 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:31.982 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:17.928 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:36.661 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:22.618 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:36.664 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:22.620 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:36.681 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:22.638 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:36.711 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:22.667 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:36.712 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:22.658 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:36.715 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:22.671 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:36.715 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:22.661 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:36.730 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:22.686 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:36.731 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:22.677 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:47.740 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:7462ddf6:::163:head 2024-01-27T03:57:47.740 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:5d165639:::164:head 2024-01-27T03:57:47.740 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:f43765fc:::165:head 2024-01-27T03:57:47.740 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:b4c720e9:::166:head 2024-01-27T03:57:47.740 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:e694b040:::167:head 2024-01-27T03:57:47.740 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:afa38db2:::168:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:77ba9f53:::169:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:87495034:::170:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:7c96bf0e:::171:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:dbe346cc:::172:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:e943ec24:::173:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:f97a9c0c:::174:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:6f26e74d:::175:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:4f95e106:::176:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:0e6f2f8f:::177:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:05db05f1:::178:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:38a78d66:::179:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:d095610b:::180:head 2024-01-27T03:57:47.741 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:a1a9d709:::181:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:1e5d39db:::182:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:f7df4fb9:::183:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:03a7f161:::184:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:ba70721e:::185:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:28e5662d:::186:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:973d52de:::187:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:4303eb1c:::188:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:b990b48e:::189:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:29b8165b:::190:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:3547f197:::191:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:7e260936:::192:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:1abec7b1:::193:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:10fdda93:::194:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:15817eea:::195:head 2024-01-27T03:57:47.742 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:770bab57:::196:head 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:ed9e13e7:::197:head 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:71471a8f:::198:head 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: checking for 211:10fb1d02:::199:head 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22783 ms) 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,0 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: first is 1706327816 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,0 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,0 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,1706327819,1706327820,0 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,1706327819,1706327820,0 2024-01-27T03:57:47.743 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,1706327819,1706327820,0 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,1706327819,1706327820,1706327822,1706327823,0 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,1706327819,1706327820,1706327822,1706327823,0 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,1706327819,1706327820,1706327822,1706327823,0 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,1706327819,1706327820,1706327822,1706327823,1706327825,1706327826,0 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,1706327819,1706327820,1706327822,1706327823,1706327825,1706327826,0 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327816,1706327819,1706327820,1706327822,1706327823,1706327825,1706327826,0 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706327820,1706327822,1706327823,1706327825,1706327826,1706327828,1706327829,0 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: first now 1706327820, trimmed 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22320 ms) 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: foo0 2024-01-27T03:57:47.744 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-01-27T03:57:47.745 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (16857 ms) 2024-01-27T03:57:47.745 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-01-27T03:57:51.703 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:37.658 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:51.705 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:37.661 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:51.729 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:37.685 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:51.749 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:37.705 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:51.752 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:37.709 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:51.755 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:37.701 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:51.758 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:37.703 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:51.778 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:37.734 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:51.781 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:37.727 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:52.751 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:38.707 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:52.752 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:38.709 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:52.763 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:38.719 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:52.799 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:38.744 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:52.800 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:38.745 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:52.808 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:38.764 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:52.809 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:38.765 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:52.810 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:38.755 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:52.824 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:38.780 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:57.924 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:43.880 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:57.926 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:43.882 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:57.945 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:43.901 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:57.978 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:43.934 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:57.979 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:43.935 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:57.982 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:43.928 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:57.984 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:43.930 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:57.998 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:43.955 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:57:58.005 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:43.951 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:06.706 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:52.661 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:06.752 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:52.709 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:06.772 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:52.728 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:06.801 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:52.757 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:06.802 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:52.758 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:06.803 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:52.749 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:06.805 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:52.751 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:06.820 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:52.776 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:06.823 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:52.769 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:11.821 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:57.777 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:11.824 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:57.780 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:11.842 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:15:57.798 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:11.875 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:57.820 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:11.877 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:57.822 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:11.889 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:57.845 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:11.890 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:15:57.836 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:11.891 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:57.848 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:11.908 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:15:57.865 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:21.986 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:07.942 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:21.990 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:07.946 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:22.016 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:07.971 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:22.038 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:07.994 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:22.043 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:07.999 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:22.054 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:08.000 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:22.057 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:08.003 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:22.068 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:08.024 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:22.086 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:08.032 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:32.033 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:17.989 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:32.090 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:18.046 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:32.123 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:18.079 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:32.153 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:18.109 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:32.154 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:18.099 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:32.165 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:18.121 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:32.166 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:18.112 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:32.193 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:18.149 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:32.195 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:18.140 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:36.916 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:22.872 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:36.918 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:22.874 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:36.932 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:22.888 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:36.962 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:22.918 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:36.965 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:22.911 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:36.965 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:22.921 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:36.966 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:22.912 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:36.979 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:22.925 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:36.980 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:22.936 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:52.023 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:37.979 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:52.073 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:38.029 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:52.081 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:38.027 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:52.086 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:38.031 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:52.103 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:38.059 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:52.121 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:38.066 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:52.139 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:38.095 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:52.141 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:38.098 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:58:52.171 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:38.127 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.055 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:58.010 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.056 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:58.013 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.081 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:58.036 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.105 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:58.051 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.105 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:58.062 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.106 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:58.052 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.107 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:58.063 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.130 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:58.075 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.133 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:58.089 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.827 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:58.783 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.828 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:58.784 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.835 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:58.781 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.837 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:58.782 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.837 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:16:58.793 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.845 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:16:58.791 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.849 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:58.805 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.850 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:58.806 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:12.859 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:16:58.815 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:27.067 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:13.023 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:27.122 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:13.068 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:27.123 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:13.079 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:27.123 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:13.079 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:27.126 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:13.072 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:27.127 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:13.083 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:27.153 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:13.109 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:27.161 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:13.118 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:27.167 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:13.113 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:32.231 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:18.184 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:32.233 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:18.189 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:32.250 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:18.206 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:32.286 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:18.242 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:32.287 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:18.244 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:32.299 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:18.256 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:32.305 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:18.250 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:32.306 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:18.252 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:32.319 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:18.265 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:42.174 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:28.130 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:42.177 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:28.133 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:42.198 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:28.155 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:42.223 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:28.179 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:42.227 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:28.183 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:42.235 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:28.181 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:42.238 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:28.184 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:42.256 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:28.209 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:42.267 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:28.213 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:52.262 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:38.218 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:52.264 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:38.220 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:52.293 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:38.249 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:52.309 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:38.265 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:52.313 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:38.270 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:52.318 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:38.264 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:52.320 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:38.265 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:52.341 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:38.297 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:52.345 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:38.290 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:57.404 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:43.360 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:57.409 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:43.365 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:57.420 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:43.377 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:57.453 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:43.409 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:57.454 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:43.411 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:57.455 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:43.400 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:57.456 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:43.402 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:57.467 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:43.423 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T03:59:57.470 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:43.416 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:12.325 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:58.281 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:12.328 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:58.284 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:12.346 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:17:58.303 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:12.377 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:58.333 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:12.378 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:58.324 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:12.379 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:58.325 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:12.380 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:58.336 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:12.395 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:17:58.341 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:12.397 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:17:58.354 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:16.435 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:02.391 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:16.440 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:02.396 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:16.464 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:02.420 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:16.492 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:02.438 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:16.493 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:02.439 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:16.509 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:02.455 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:16.515 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:02.471 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:16.518 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:02.474 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:16.532 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:02.489 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:17.468 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:03.424 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:17.472 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:03.428 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:17.475 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:03.421 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:17.478 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:03.434 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:17.480 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:03.436 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:17.540 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:03.485 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:17.549 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:03.506 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:17.557 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:03.514 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:17.575 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:03.521 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:32.434 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:18.390 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:32.436 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:18.392 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:32.469 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:18.425 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:32.481 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:18.426 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:32.482 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:18.428 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:32.498 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:18.455 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:32.500 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:18.456 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:32.503 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:18.449 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:32.521 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:18.478 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:33.314 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:19.260 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:33.317 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:19.263 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:33.330 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:19.276 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:33.376 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:19.332 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:33.379 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:19.334 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:33.388 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:19.344 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:33.436 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:19.392 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:33.437 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:19.393 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:33.447 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:19.403 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:47.563 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:33.519 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:47.566 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:33.522 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:47.594 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:33.551 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:47.617 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:33.573 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:47.617 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:33.563 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:47.618 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:33.574 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:47.619 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:33.564 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:47.647 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:33.603 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:47.697 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:33.642 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:52.708 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:38.664 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:52.711 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:38.667 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:52.728 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:38.684 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:52.770 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:38.726 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:52.774 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:38.730 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:52.775 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:38.721 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:52.777 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:38.723 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:52.791 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:38.747 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:00:52.792 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:38.738 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:02.633 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:48.589 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:02.634 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:48.591 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:02.651 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:48.607 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:02.682 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:48.628 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:02.686 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:48.643 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:02.687 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:48.632 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:02.688 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:48.645 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:02.704 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:48.661 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:02.705 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:48.650 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:11.677 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T04:01:11.668+0000 21e7c640 -1 snap_mapper.add_oid found existing snaps mapped on 229:f9172301:test-rados-api-smithi110-40469-80::foo:2, removing 2024-01-27T04:01:11.693 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T04:01:11.684+0000 2c691640 -1 snap_mapper.add_oid found existing snaps mapped on 229:f9172301:test-rados-api-smithi110-40469-80::foo:2, removing 2024-01-27T04:01:12.808 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:58.764 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:12.812 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:58.768 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:12.831 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:18:58.787 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:12.870 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:58.816 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:12.873 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:58.819 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:12.880 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:58.836 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:12.883 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:58.840 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:12.896 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:18:58.842 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:12.908 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:18:58.864 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:17.748 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:03.704 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:17.753 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:03.709 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:17.769 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:03.726 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:17.807 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:03.753 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:17.811 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:03.757 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:17.813 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:03.770 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:17.816 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:03.773 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:17.831 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:03.776 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:17.835 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:03.791 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:32.777 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:18.733 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:32.779 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:18.735 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:32.812 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:18.769 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:32.826 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:18.772 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:32.830 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:18.776 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:32.831 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:18.787 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:32.837 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:18.793 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:32.864 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:18.810 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:32.871 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:18.827 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:33.689 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:19.635 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:33.690 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:19.636 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:33.699 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:19.645 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:33.734 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:19.690 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:33.735 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:19.691 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:33.744 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:19.700 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:33.765 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:19.722 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:33.767 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:19.723 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:33.776 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:19.732 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:47.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20913 ms) 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (26162 ms) 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (5238 ms) 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (4888 ms) 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (5450 ms) 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4680 ms) 2024-01-27T04:01:47.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5332 ms) 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (41131 ms) 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17766 ms) 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5491 ms) 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-01-27T04:01:47.649 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4541 ms) 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27942 ms) 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16641 ms) 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (7444 ms) 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.650 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5623 ms) 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: waiting for scrubs... 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: done waiting 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26548 ms) 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12539 ms) 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11257 ms) 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:01:47.651 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11234 ms) 2024-01-27T04:01:48.136 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:34.092 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:48.142 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:34.098 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:48.200 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:34.156 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:48.202 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:34.148 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:48.204 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:34.150 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:48.224 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:34.180 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:48.226 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:34.183 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:48.247 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:34.193 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:48.295 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:34.251 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:49.163 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:35.109 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:49.164 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:35.110 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:49.174 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:35.119 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:49.253 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:35.209 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:49.254 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:35.210 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:49.263 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:35.220 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:49.297 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:35.253 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:49.298 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:35.255 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:49.307 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:35.264 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:53.224 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:39.180 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:53.229 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:39.185 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:53.248 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:39.204 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:53.276 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:39.221 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:53.279 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:39.225 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:53.290 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:39.246 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:53.295 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:39.252 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:53.297 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:39.243 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:01:53.319 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:39.276 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:03.131 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:49.087 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:03.135 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:49.091 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:03.160 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:49.116 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:03.182 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:49.128 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:03.183 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:49.129 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:03.188 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:49.144 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:03.190 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:49.147 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:03.207 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:49.152 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:03.228 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:49.184 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:13.323 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:59.279 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:13.328 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:59.284 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:13.350 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:19:59.307 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:13.378 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:59.335 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:13.379 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:59.336 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:13.386 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:59.332 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:13.388 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:59.334 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:13.404 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:19:59.361 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:13.413 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:19:59.359 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:18.150 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:04.107 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:18.200 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:04.156 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:18.200 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:04.146 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:18.204 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:04.150 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:18.220 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:04.165 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:18.221 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:04.177 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:18.279 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:04.235 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:18.283 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:04.239 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:18.300 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:04.257 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:31.658 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:17.614 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:31.670 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:17.627 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:31.672 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:17.618 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:31.674 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:17.619 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:31.706 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:17.662 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:31.721 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:17.678 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:31.733 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:17.689 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:31.747 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:17.703 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:31.748 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:17.694 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:33.356 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:19.312 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:33.405 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:19.350 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:33.406 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:19.352 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:33.413 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:19.369 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:33.415 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:19.360 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:33.415 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:19.372 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:33.417 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:19.374 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:33.427 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:19.383 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:33.431 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:19.387 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:38.377 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:24.333 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:38.379 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:24.335 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:38.400 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:24.356 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:38.431 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:24.376 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:38.432 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:24.378 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:38.442 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:24.398 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:38.444 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:24.400 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:38.448 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:24.394 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:38.469 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:24.425 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:53.345 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:39.301 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:53.349 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:39.305 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:53.372 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:39.328 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:53.394 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:39.351 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:53.396 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:39.342 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:53.397 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:39.353 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:53.400 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:39.346 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:53.424 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:39.380 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:53.426 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:39.372 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:54.053 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:39.998 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:54.053 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:39.999 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:54.062 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:40.008 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:54.102 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:40.058 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:54.103 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:40.060 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:54.112 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:40.068 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:54.135 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:40.091 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:54.136 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:40.092 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:54.144 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:40.101 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:02:56.263 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:02:56.247+0000 2227d640 -1 snap_mapper.add_oid found existing snaps mapped on 103:7a571726:test-rados-api-smithi110-40469-97::foo:21, removing 2024-01-27T04:02:56.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-27T04:02:56.277+0000 27a88640 -1 snap_mapper.add_oid found existing snaps mapped on 103:7a571726:test-rados-api-smithi110-40469-97::foo:21, removing 2024-01-27T04:03:08.559 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:54.515 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:08.562 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:54.519 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:08.594 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:54.550 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:08.611 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:54.557 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:08.615 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:54.561 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:08.627 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:54.583 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:08.631 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:54.587 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:08.649 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:54.595 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:08.669 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:54.625 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:13.572 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:59.528 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:13.574 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:59.530 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:13.593 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:20:59.549 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:13.618 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:59.575 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:13.622 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:59.578 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:13.628 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:59.573 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:13.631 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:59.577 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:13.637 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:20:59.594 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:13.648 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:20:59.594 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:23.504 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:09.460 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:23.508 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:09.464 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:23.540 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:09.497 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:23.551 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:09.497 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:23.553 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:09.498 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:23.570 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:09.526 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:23.571 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:09.528 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:23.585 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:09.531 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:23.596 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:09.552 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:33.494 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:19.450 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:33.563 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:19.508 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:33.567 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:19.512 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:33.567 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:19.523 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:33.601 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:19.546 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:33.606 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:19.562 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:33.613 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:19.569 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:33.638 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:19.594 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:33.672 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:19.628 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:34.333 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:20.278 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:34.339 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:20.285 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:34.416 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:20.361 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:34.497 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:20.453 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:34.501 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:20.458 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:34.559 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:20.515 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:34.680 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:20.637 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:34.682 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:20.638 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:34.697 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:20.653 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:38.579 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:24.535 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:38.580 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:24.536 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:38.593 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:24.549 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:38.617 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:24.563 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:38.621 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:24.577 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:38.622 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:24.568 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:38.626 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:24.583 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:38.638 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:24.584 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:38.640 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:24.597 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:48.624 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:34.580 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:48.629 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:34.585 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:48.664 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:34.620 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:48.700 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:34.656 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:48.703 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:34.660 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:48.704 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:34.650 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:48.709 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:34.655 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:48.736 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:34.692 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:48.737 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:34.683 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:53.578 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:39.533 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:53.582 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:39.538 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:53.599 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:39.555 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:53.627 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:39.572 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:53.629 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:39.575 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:53.644 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:39.590 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:53.645 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:39.601 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:53.649 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:39.605 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:03:53.670 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:39.626 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:03.715 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:49.671 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:03.764 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:49.720 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:03.782 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:49.727 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:03.786 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:49.732 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:03.792 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:49.748 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:03.813 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:49.759 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:03.826 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:49.782 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:03.827 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:49.784 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:03.854 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:49.810 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:13.707 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:59.663 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:13.711 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:59.668 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:13.737 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:21:59.693 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:13.758 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:59.704 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:13.761 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:59.707 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:13.774 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:59.730 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:13.777 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:59.733 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:13.787 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:21:59.733 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:13.805 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:21:59.761 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:18.723 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:04.679 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:18.726 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:04.682 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:18.750 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:04.707 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:18.780 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:04.737 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:18.783 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:04.729 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:18.784 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:04.740 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:18.784 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:04.730 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:18.799 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:04.756 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:18.800 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:04.745 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:32.645 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:18.600 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:32.659 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:18.615 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:32.662 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:18.608 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:32.665 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:18.611 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:32.668 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:18.625 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:32.671 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:18.627 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:32.760 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:18.705 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:32.773 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:18.730 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:32.777 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:18.733 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:34.015 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:19.972 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:34.017 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:19.974 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:34.029 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:19.985 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:34.065 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:20.011 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:34.067 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:20.013 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:34.079 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:20.024 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:34.086 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:20.042 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:34.088 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:20.045 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:34.103 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:20.060 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:48.986 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:34.942 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:48.988 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:34.944 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:49.016 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:34.973 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:49.036 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:34.993 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:49.038 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:34.983 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:49.038 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:34.994 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:49.039 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:34.985 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:49.064 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:35.010 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:49.065 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:35.022 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:58.994 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:44.950 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:58.998 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:44.955 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:59.010 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:44.967 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:59.042 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:44.998 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:59.043 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:44.989 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:59.044 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:45.000 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:59.050 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:44.995 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:59.057 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:45.013 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:04:59.064 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:45.010 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:09.212 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:55.169 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:09.214 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:55.171 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:09.239 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:22:55.196 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:09.266 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:55.211 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:09.270 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:55.226 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:09.272 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:55.217 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:09.274 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:55.230 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:09.297 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:22:55.243 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:09.301 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:22:55.257 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:19.307 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:05.264 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:19.310 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:05.266 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:19.323 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:05.269 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:19.325 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:05.271 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:19.348 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:05.304 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:19.367 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:05.312 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:19.383 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:05.339 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:19.386 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:05.342 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:19.430 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:05.387 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:24.153 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:10.109 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:24.158 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:10.114 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:24.172 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:10.128 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:24.200 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:10.146 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:24.203 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:10.149 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:24.204 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:10.161 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:24.208 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:10.164 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:24.217 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:10.163 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:24.222 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:10.178 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:39.214 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:25.170 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:39.218 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:25.175 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:39.255 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:25.211 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:39.270 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:25.216 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:39.272 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:25.229 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:39.276 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:25.232 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:39.277 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:25.223 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:39.309 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:25.266 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:39.310 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:25.256 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:40.109 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:26.054 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:40.110 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:26.056 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:40.121 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:26.067 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:40.130 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:26.086 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:40.131 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:26.087 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:40.140 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:26.097 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:40.188 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:26.144 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:40.189 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:26.146 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:40.202 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:26.158 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:54.422 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:40.378 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:54.425 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:40.381 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:54.465 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:40.422 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:54.474 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:40.430 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:54.474 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:40.420 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:54.478 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:40.424 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:54.480 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:40.436 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:54.509 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:40.455 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:54.516 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:40.472 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:59.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-01-27T04:05:59.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:05:59.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11533 ms) 2024-01-27T04:05:59.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-01-27T04:05:59.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:05:59.390 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8623 ms) 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6568 ms) 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (27766 ms) 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15873 ms) 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12273 ms) 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: 00000010 69 73 |is| 2024-01-27T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: 00000012 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (14389 ms) 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: require_osd_release = squid 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7511 ms) 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (952019 ms total) 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1536 ms) 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11341 ms) 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (21739 ms) 2024-01-27T04:05:59.392 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (30 ms) 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (34646 ms total) 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (9006 ms) 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9889 ms) 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: waiting for scrub... 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: done waiting 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (27021 ms) 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11777 ms) 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-01-27T04:05:59.393 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (10945 ms) 2024-01-27T04:05:59.394 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-01-27T04:05:59.394 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (9742 ms) 2024-01-27T04:05:59.394 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-01-27T04:05:59.394 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12858 ms) 2024-01-27T04:05:59.394 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-01-27T04:05:59.492 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:45.448 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:59.495 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:45.451 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:59.515 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:45.471 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:59.546 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:45.491 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:59.548 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:45.494 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:59.553 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:45.509 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:59.555 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:45.511 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:59.563 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:45.509 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:05:59.573 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:45.529 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:09.531 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:55.487 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:09.534 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:55.490 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:09.561 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:23:55.517 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:09.577 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:55.523 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:09.578 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:55.524 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:09.588 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:55.544 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:09.592 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:55.548 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:09.600 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:23:55.546 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:09.614 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:23:55.570 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:19.452 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:05.407 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:19.454 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:05.410 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:19.482 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:05.438 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:19.502 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:05.447 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:19.507 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:05.453 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:19.508 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:05.464 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:19.511 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:05.467 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:19.532 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:05.477 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:19.538 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:05.495 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:29.459 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:15.415 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:29.461 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:15.417 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:29.485 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:15.441 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:29.504 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:15.450 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:29.511 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:15.467 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:29.512 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:15.469 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:29.525 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:15.470 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:29.549 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:15.506 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:29.550 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:15.495 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:33.767 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:19.723 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:33.770 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:19.727 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:33.788 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:19.744 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:33.821 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:19.767 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:33.822 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:19.778 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:33.823 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:19.768 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:33.823 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:19.780 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:33.842 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:19.787 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:33.842 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:19.798 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:39.591 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:25.547 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:39.595 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:25.551 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:39.617 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:25.574 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:39.650 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:25.595 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:39.651 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:25.597 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:39.668 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:25.614 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:39.686 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:25.642 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:39.691 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:25.647 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:39.709 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:25.665 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:44.703 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:30.659 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:44.705 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:30.662 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:44.721 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:30.677 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:44.748 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:30.694 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:44.750 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:30.696 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:44.754 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:30.710 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:44.755 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:30.712 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:44.768 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:30.714 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:44.772 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:30.728 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:59.628 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:45.584 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:59.630 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:45.586 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:59.664 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:45.620 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:59.688 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:45.644 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:59.690 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:45.646 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:59.700 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:45.646 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:59.718 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:45.664 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:59.736 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:45.692 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:06:59.838 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:45.784 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:00.504 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:46.460 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:00.505 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:46.461 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:00.511 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:46.468 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:00.513 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:46.469 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:00.514 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:24:46.471 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:00.517 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:46.463 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:00.518 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:46.464 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:00.527 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:24:46.473 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:00.537 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:24:46.493 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:14.670 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:00.626 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:14.674 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:00.630 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:14.710 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:00.666 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:14.723 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:00.668 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:14.724 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:00.680 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:14.726 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:00.672 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:14.727 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:00.683 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:14.757 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:00.702 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:14.763 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:00.719 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:19.670 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:05.626 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:19.675 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:05.631 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:19.690 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:05.646 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:19.722 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:05.668 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:19.722 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:05.678 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:19.723 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:05.669 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:19.725 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:05.681 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:19.737 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:05.683 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:19.740 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:05.696 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:34.729 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:20.685 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:34.735 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:20.691 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:34.767 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:20.723 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:34.771 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:20.727 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:34.802 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:20.748 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:34.808 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:20.754 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:34.816 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:20.773 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:34.842 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:20.787 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:34.849 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:20.805 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:39.749 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:25.705 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:39.807 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:25.762 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:39.851 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:25.797 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:39.853 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:25.798 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:39.865 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:25.821 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:39.865 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:25.821 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:39.868 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:25.824 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:39.893 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:25.839 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:39.917 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:25.873 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:49.751 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:35.707 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:49.800 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:35.746 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:49.801 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:35.747 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:49.818 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:35.774 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:49.832 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:35.778 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:49.862 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:35.818 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:49.880 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:35.836 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:49.883 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:35.839 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:49.912 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:35.869 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:59.915 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:45.871 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:07:59.964 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:45.920 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:00.003 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:45.959 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:00.014 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:45.960 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:00.019 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:45.964 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:00.032 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:45.988 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:00.034 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:45.990 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:00.059 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:46.005 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:00.064 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:46.020 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:04.856 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:50.812 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:04.857 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:50.814 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:04.871 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:25:50.828 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:04.903 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:50.859 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:04.904 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:50.850 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:04.905 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:50.851 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:04.908 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:50.864 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:04.915 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:25:50.861 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:04.919 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:25:50.876 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:19.931 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:05.887 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:19.934 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:05.890 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:19.964 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:05.920 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:19.981 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:05.926 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:19.982 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:05.928 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:19.986 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:05.942 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:19.987 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:05.943 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.013 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:05.969 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.018 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:05.964 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.875 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:06.820 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.876 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:06.821 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.885 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:06.830 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.893 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:06.850 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.894 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:06.851 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.903 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:06.860 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.926 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:06.882 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.927 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:06.883 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:20.936 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:06.892 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:34.966 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:20.922 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:34.971 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:20.927 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:34.998 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:20.954 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:35.019 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:20.965 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:35.021 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:20.967 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:35.032 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:20.988 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:35.035 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:20.991 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:35.044 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:20.990 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:35.059 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:21.015 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:40.199 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:26.145 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:40.201 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:26.146 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:40.206 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:26.163 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:40.209 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:26.165 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:40.230 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:26.186 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:40.237 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:26.182 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:40.280 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:26.236 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:40.285 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:26.241 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:40.307 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:26.263 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:41.327 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:27.283 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:41.399 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:27.355 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:41.418 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:27.374 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:41.441 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:27.397 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:41.465 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:27.422 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:41.471 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:27.417 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:41.474 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:27.419 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:41.492 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:27.437 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:41.499 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:27.455 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:50.128 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:36.083 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:50.132 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:36.086 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:50.155 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:36.111 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:50.181 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:36.127 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:50.183 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:36.129 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:50.184 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:36.141 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:50.186 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:36.143 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:50.207 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:36.152 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:08:50.215 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:36.171 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:00.282 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:46.238 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:00.285 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:46.242 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:00.311 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:46.267 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:00.332 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:46.278 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:00.333 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:46.289 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:00.333 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:46.279 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:00.334 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:46.290 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:00.357 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:46.303 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:00.359 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:46.315 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:01.218 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10155 ms) 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14428 ms) 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9610 ms) 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (16119 ms) 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8840 ms) 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9927 ms) 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: hmm, no HitSet yet 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: ok, hit_set contains 261:602f83fe:::foo:head 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10012 ms) 2024-01-27T04:09:01.219 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: first is 1706328437 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,1706328439,1706328440,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,1706328439,1706328440,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,1706328439,1706328440,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,1706328439,1706328440,1706328442,1706328443,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,1706328439,1706328440,1706328442,1706328443,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,1706328439,1706328440,1706328442,1706328443,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,1706328439,1706328440,1706328442,1706328443,1706328445,1706328446,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,1706328439,1706328440,1706328442,1706328443,1706328445,1706328446,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328437,1706328439,1706328440,1706328442,1706328443,1706328445,1706328446,0 2024-01-27T04:09:01.220 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: got ls 1706328440,1706328442,1706328443,1706328445,1706328446,1706328448,1706328449,0 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: first now 1706328440, trimmed 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (23877 ms) 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: foo0 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (19618 ms) 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (21778 ms) 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (26082 ms) 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5411 ms) 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4981 ms) 2024-01-27T04:09:01.221 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-01-27T04:09:01.222 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4396 ms) 2024-01-27T04:09:01.222 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-01-27T04:09:01.222 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3346 ms) 2024-01-27T04:09:01.222 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (279818 ms total) 2024-01-27T04:09:01.222 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: 2024-01-27T04:09:01.222 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-01-27T04:09:01.222 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1451729 ms total) 2024-01-27T04:09:01.222 INFO:tasks.workunit.client.0.smithi110.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-01-27T04:09:01.224 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.224 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40107 2024-01-27T04:09:01.224 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40107 2024-01-27T04:09:01.224 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.224 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40358 2024-01-27T04:09:01.224 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40358 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40821 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40821 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40549 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40549 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40431 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40431 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40192 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40192 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40490 2024-01-27T04:09:01.225 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40490 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40953 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40953 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40992 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40992 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40225 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40225 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40859 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40859 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40753 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40753 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.226 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40080 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40080 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40144 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40144 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40772 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40772 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40071 2024-01-27T04:09:01.227 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40071 2024-01-27T04:09:05.215 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:51.171 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:05.217 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:51.173 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:05.232 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:51.188 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:05.268 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:51.224 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:05.268 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:51.213 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:05.269 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:51.225 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:05.270 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:51.215 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:05.284 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:51.240 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:05.284 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:51.229 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:10.257 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:56.213 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:10.259 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:56.215 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:10.276 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:26:56.233 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:10.315 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:56.261 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:10.316 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:56.273 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:10.317 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:56.263 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:10.317 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:56.274 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:10.334 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:26:56.280 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:10.335 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:26:56.292 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:25.283 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:11.239 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:25.286 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:11.243 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:25.309 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:11.264 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:25.337 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:11.294 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:25.338 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:11.283 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:25.340 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:11.286 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:25.342 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:11.299 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:25.356 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:11.301 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:25.365 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:11.321 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:26.025 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:11.971 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:26.026 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:11.972 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:26.034 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:11.980 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:26.089 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:12.045 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:26.089 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:12.046 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:26.090 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:12.046 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:26.091 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:12.047 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:26.098 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:12.054 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:26.099 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:12.055 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:45.332 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:31.288 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:45.335 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:31.292 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:45.357 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:31.313 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:45.389 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:31.335 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:45.393 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:31.338 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:45.397 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:31.353 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:45.400 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:31.356 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:45.413 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:31.358 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:45.417 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:31.373 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:46.100 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:32.045 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:46.100 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:32.046 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:46.109 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:32.054 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:46.122 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:32.079 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:46.123 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:32.079 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:46.124 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:32.080 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:46.124 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:32.080 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:46.132 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:32.088 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:09:46.132 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:32.088 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:05.401 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:51.357 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:05.403 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:51.359 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:05.420 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:51.377 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:05.453 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:51.398 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:05.460 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:51.405 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:05.465 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:51.421 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:05.468 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:51.425 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:05.479 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:51.424 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:05.487 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:51.443 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:06.109 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:52.054 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:06.110 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:52.055 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:06.118 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:27:52.063 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:06.133 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:52.089 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:06.134 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:52.090 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:06.144 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:27:52.100 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:06.145 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:52.102 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:06.147 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:52.103 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:06.154 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:27:52.111 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:25.450 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:11.406 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:25.452 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:11.409 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:25.470 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:11.427 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:25.500 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:11.456 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:25.502 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:11.458 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:25.504 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:11.449 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:25.508 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:11.454 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:25.525 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:11.471 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:25.528 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:11.484 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:26.164 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:12.109 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:26.165 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:12.110 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:26.173 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:12.119 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:26.184 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:12.140 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:26.185 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:12.142 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:26.194 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:12.150 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:26.194 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:12.150 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:26.195 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:12.151 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:26.203 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:12.159 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:45.518 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:31.473 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:45.520 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:31.476 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:45.539 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:31.495 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:45.569 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:31.514 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:45.570 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:31.516 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:45.575 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:31.531 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:45.576 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:31.533 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:45.588 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:31.533 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:45.599 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:31.555 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:46.271 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:32.216 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:46.272 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:32.218 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:46.280 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:32.226 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:46.305 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:32.262 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:46.307 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:32.263 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:46.319 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:32.275 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:46.333 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:32.289 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:46.334 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:32.290 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:10:46.342 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:32.299 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:05.521 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:51.477 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:05.568 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:51.524 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:05.585 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:51.542 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:05.614 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:51.559 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:05.615 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:51.571 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:05.618 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:51.564 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:05.620 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:51.576 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:05.634 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:51.580 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:05.637 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:51.594 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:06.269 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:52.215 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:06.271 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:52.216 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:06.285 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:28:52.231 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:06.343 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:52.299 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:06.366 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:52.322 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:06.368 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:52.324 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:06.379 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:28:52.335 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:06.395 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:52.351 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:06.404 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:28:52.361 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:25.621 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:11.577 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:25.668 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:11.613 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:25.668 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:11.624 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:25.672 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:11.618 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:25.690 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:11.646 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:25.691 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:11.636 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:25.716 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:11.672 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:25.719 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:11.675 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:25.736 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:11.692 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:26.379 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:12.324 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:26.380 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:12.325 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:26.391 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:12.337 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:26.455 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:12.411 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:26.456 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:12.413 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:26.466 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:12.422 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:26.467 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:12.424 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:26.468 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:12.425 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:26.477 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:12.434 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:43.231 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:29.186 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:43.283 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:29.239 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:43.304 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:29.259 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:43.329 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:29.274 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:43.332 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:29.278 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:43.350 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:29.306 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:43.351 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:29.307 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:43.353 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:29.299 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:43.372 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:29.328 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:45.671 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:31.627 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:45.718 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:31.674 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:45.730 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:31.686 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:45.764 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:31.710 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:45.767 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:31.713 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:45.771 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:31.727 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:45.774 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:31.730 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:45.777 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:31.722 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:45.784 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:31.740 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:46.327 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:32.273 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:46.328 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:32.274 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:46.336 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:32.281 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:46.345 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:32.301 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:46.346 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:32.302 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:46.354 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:32.310 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:46.367 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:32.323 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:46.368 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:32.324 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:11:46.377 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:32.333 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:05.880 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:51.836 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:05.890 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:51.846 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:05.928 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:51.884 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:05.938 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:51.883 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:05.941 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:51.887 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:05.964 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:51.921 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:05.966 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:51.911 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:05.966 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:51.923 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:05.991 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:51.947 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:06.564 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:52.520 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:06.565 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:52.521 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:06.567 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:52.513 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:06.568 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:52.514 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:06.574 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:29:52.531 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:06.577 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:29:52.523 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:06.605 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:52.561 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:06.606 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:52.563 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:06.616 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:29:52.573 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:25.919 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:11.875 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:25.920 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:11.877 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:25.939 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:11.895 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:25.967 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:11.910 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:25.968 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:11.912 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:25.968 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:11.924 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:25.973 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:11.929 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:25.983 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:11.929 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:25.991 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:11.947 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:26.567 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:12.513 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:26.568 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:12.514 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:26.572 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:12.528 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:26.575 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:12.530 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:26.577 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:12.522 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:26.582 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:12.539 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:26.599 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:12.555 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:26.600 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:12.557 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:26.610 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:12.566 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:45.964 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:31.920 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:45.966 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:31.922 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:45.987 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:31.944 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.007 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:31.952 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.010 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:31.955 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.014 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:31.970 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.015 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:31.971 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.029 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:31.974 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.033 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:31.989 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.421 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:32.367 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.424 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:32.380 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.425 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:32.371 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.437 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:32.393 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.438 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:32.394 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.446 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:32.403 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.479 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:32.435 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.483 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:32.429 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:12:46.492 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:32.448 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.011 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:51.967 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.014 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:51.970 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.040 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:51.997 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.057 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:52.002 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.060 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:52.006 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.068 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:52.024 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.072 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:52.028 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.078 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:52.023 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.090 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:52.047 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.677 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:52.623 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.678 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:52.624 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.682 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:52.638 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.683 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:52.640 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.687 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:30:52.632 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.693 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:30:52.649 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.707 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:52.662 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.707 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:52.663 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:06.716 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:30:52.672 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.080 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:12.035 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.083 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:12.040 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.117 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:12.073 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.136 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:12.092 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.139 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:12.095 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.157 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:12.113 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.163 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:12.109 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.164 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:12.110 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.181 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:12.126 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.715 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:12.671 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.718 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:12.674 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.726 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:12.682 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.731 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:12.677 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.733 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:12.678 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.737 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:12.694 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.739 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:12.695 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.740 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:12.686 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:26.747 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:12.703 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:46.104 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:32.060 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:46.108 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:32.064 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:46.127 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:32.083 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:46.157 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:32.113 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:46.160 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:32.117 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:46.168 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:32.114 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:46.175 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:32.121 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:46.183 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:32.139 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:46.192 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:32.138 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:51.076 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:37.032 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:51.122 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:37.068 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:51.123 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:37.079 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:51.125 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:37.071 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:51.137 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:37.093 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:51.138 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:37.084 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:51.169 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:37.125 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:51.172 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:37.128 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:13:51.184 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:37.140 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:06.157 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:52.113 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:06.160 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:52.116 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:06.175 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:52.131 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:06.205 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:52.151 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:06.206 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:52.152 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:06.208 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:52.164 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:06.210 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:52.166 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:06.220 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:52.165 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:06.230 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:52.186 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:11.166 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:57.122 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:11.170 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:57.126 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:11.184 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:31:57.140 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:11.214 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:57.170 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:11.215 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:57.160 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:11.217 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:57.162 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:11.218 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:57.174 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:11.228 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:31:57.173 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:11.229 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:31:57.186 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:26.202 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:12.158 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:26.204 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:12.160 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:26.220 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:12.177 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:26.252 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:12.198 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:26.254 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:12.210 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:26.255 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:12.201 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:26.255 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:12.212 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:26.271 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:12.216 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:26.271 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:12.227 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:31.214 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:17.170 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:31.217 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:17.173 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:31.233 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:17.189 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:31.261 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:17.207 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:31.264 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:17.209 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:31.269 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:17.225 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:31.272 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:17.228 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:31.275 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:17.221 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:31.282 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:17.238 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:46.261 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:32.217 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:46.265 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:32.222 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:46.282 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:32.238 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:46.312 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:32.258 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:46.315 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:32.261 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:46.326 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:32.283 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:46.328 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:32.284 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:46.330 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:32.276 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:46.344 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:32.301 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:51.277 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:37.233 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:51.280 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:37.237 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:51.294 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:37.250 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:51.327 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:37.273 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:51.328 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:37.284 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:51.329 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:37.275 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:51.332 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:37.288 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:51.340 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:37.286 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:51.343 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:37.299 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:58.595 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:44.551 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:58.599 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:44.555 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:58.605 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:44.561 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:58.606 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:44.551 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:58.608 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:44.554 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:58.609 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:44.565 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:58.622 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:44.578 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:58.627 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:44.572 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:14:58.630 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:44.586 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:06.310 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:52.266 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:06.314 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:52.270 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:06.329 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:52.285 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:06.358 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:52.304 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:06.361 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:52.317 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:06.362 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:52.308 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:06.364 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:52.320 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:06.375 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:52.321 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:06.377 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:52.333 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:11.440 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:57.396 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:11.494 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:57.451 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:11.510 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:32:57.466 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:11.539 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:57.484 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:11.544 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:57.490 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:11.547 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:57.503 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:11.549 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:57.505 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:11.561 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:32:57.506 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:11.566 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:32:57.522 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:21.561 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:07.517 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:21.566 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:07.522 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:21.603 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:07.559 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:21.630 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:07.575 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:21.632 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:07.588 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:21.638 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:07.584 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:21.639 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:07.595 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:21.667 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:07.623 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:21.668 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:07.614 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:31.475 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:17.431 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:31.477 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:17.433 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:31.525 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:17.481 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:31.526 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:17.472 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:31.529 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:17.475 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:31.533 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:17.490 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:31.535 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:17.492 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:31.553 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:17.499 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:31.562 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:17.518 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:32.928 INFO:tasks.workunit.client.0.smithi110.stdout: misc: Running main() from gmock_main.cc 2024-01-27T04:15:32.928 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-01-27T04:15:32.928 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [----------] Global test environment set-up. 2024-01-27T04:15:32.928 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-01-27T04:15:32.928 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-01-27T04:15:32.928 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.Version (6704 ms) 2024-01-27T04:15:32.928 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-01-27T04:15:32.928 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (3592 ms) 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.LongName (1804368 ms) 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3579 ms) 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3371 ms) 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3392 ms) 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.Exec (2779 ms) 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3207 ms) 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2933 ms) 2024-01-27T04:15:32.929 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3348 ms) 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2241 ms) 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3622 ms) 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [----------] 12 tests from NeoRadosMisc (1843136 ms total) 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [----------] Global test environment tear-down 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [==========] 12 tests from 1 test suite ran. (1843136 ms total) 2024-01-27T04:15:32.930 INFO:tasks.workunit.client.0.smithi110.stdout: misc: [ PASSED ] 12 tests. 2024-01-27T04:15:36.708 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:22.664 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:36.716 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:22.672 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:36.744 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:22.700 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:36.771 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:22.717 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:36.781 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:22.737 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:36.795 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:22.751 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:36.806 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:22.752 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:36.818 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:22.774 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:36.831 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:22.776 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:51.589 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:37.544 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:51.592 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:37.548 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:51.623 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:37.579 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:51.638 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:37.583 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:51.641 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:37.587 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:51.650 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:37.606 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:51.651 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:37.608 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:51.670 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:37.616 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:51.681 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:37.638 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:52.277 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:38.223 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:52.329 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:38.275 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:52.361 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:38.307 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:52.433 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:38.389 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:52.437 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:38.393 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:52.450 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:38.406 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:52.535 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:38.489 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:52.535 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:38.491 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:15:52.548 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:38.504 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:06.627 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:52.582 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:06.629 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:52.586 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:06.659 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:52.615 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:06.675 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:52.621 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:06.676 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:52.622 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:06.678 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:52.635 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:06.682 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:52.638 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:06.704 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:52.649 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:06.710 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:52.666 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:11.861 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:57.817 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:11.867 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:57.823 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:11.888 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:33:57.844 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:11.931 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:57.876 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:11.942 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:57.888 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:11.948 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:57.904 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:11.950 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:57.907 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:11.962 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:33:57.907 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:11.973 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:33:57.929 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:21.740 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:07.696 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:21.742 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:07.699 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:21.764 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:07.720 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:21.784 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:07.730 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:21.788 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:07.733 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:21.789 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:07.746 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:21.791 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:07.747 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:21.811 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:07.756 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:21.816 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:07.772 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:22.551 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:08.497 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:22.553 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:08.499 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:22.563 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:08.509 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:22.611 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:08.567 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:22.638 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:08.594 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:22.648 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:08.604 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:22.659 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:08.615 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:22.660 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:08.616 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:22.670 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:08.627 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:36.941 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:22.897 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:36.999 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:22.955 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:37.006 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:22.952 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:37.011 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:22.957 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:37.040 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:22.996 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:37.048 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:22.993 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:37.062 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:23.019 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:37.064 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:23.021 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:37.106 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:23.063 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6667 ms) 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (1827748 ms) 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6799 ms) 2024-01-27T04:16:50.644 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3604 ms) 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3461 ms) 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3326 ms) 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3288 ms) 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3547 ms) 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3310 ms) 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3515 ms) 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-01-27T04:16:50.645 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3338 ms) 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3336 ms) 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2496 ms) 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3587 ms) 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3258 ms) 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3301 ms) 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3852 ms) 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-01-27T04:16:50.646 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3365 ms) 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3479 ms) 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3548 ms) 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3254 ms) 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3252 ms) 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3421 ms) 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3280 ms) 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-01-27T04:16:50.647 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2591 ms) 2024-01-27T04:16:50.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-01-27T04:16:50.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3340 ms) 2024-01-27T04:16:50.648 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-01-27T04:16:52.052 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:38.008 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:52.108 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:38.064 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:52.138 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:38.094 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:52.157 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:38.113 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:52.159 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:38.115 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:52.164 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:38.110 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:52.166 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:38.112 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:52.195 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:38.141 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:52.197 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:38.153 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:54.086 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:40.042 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:54.088 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:40.045 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:54.099 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:40.045 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:54.101 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:40.058 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:54.104 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:40.050 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:54.106 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:40.062 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:54.164 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:40.120 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:54.175 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:40.120 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:54.188 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:40.144 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:56.964 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:42.920 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:56.965 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:42.921 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:56.976 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:42.932 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:57.005 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:42.961 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:57.009 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:42.965 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:57.011 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:42.957 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:57.012 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:42.958 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:57.023 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:42.977 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:16:57.023 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:42.969 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:06.942 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:52.898 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:06.992 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:52.949 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:07.015 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:34:52.971 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:07.040 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:52.996 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:07.041 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:52.987 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:07.043 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:52.989 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:07.044 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:53.000 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:07.063 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:34:53.009 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:07.067 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:34:53.022 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:37.167 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:23.123 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:37.171 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:23.127 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:37.215 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:23.160 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:37.216 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:23.162 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:37.220 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:23.176 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:37.222 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:23.178 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:37.230 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:23.186 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:37.268 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:23.213 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:37.281 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:23.237 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:38.089 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:24.035 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:38.090 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:24.036 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:38.100 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:24.045 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:38.135 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:24.092 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:38.137 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:24.093 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:38.146 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:24.103 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:38.154 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:24.111 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:38.156 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:24.112 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:38.165 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:24.121 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:52.258 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:38.214 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:52.262 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:38.218 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:52.292 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:38.248 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:52.307 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:38.253 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:52.308 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:38.254 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:52.311 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:38.268 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:52.314 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:38.271 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:52.337 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:38.282 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:52.343 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:38.300 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:57.222 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:43.178 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:57.225 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:43.181 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:57.249 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:43.205 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:57.266 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:43.211 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:57.269 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:43.214 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:57.274 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:43.230 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:57.280 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:43.236 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:57.283 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:43.229 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:17:57.296 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:43.252 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:07.263 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:53.218 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:07.268 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:53.224 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:07.294 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:35:53.250 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:07.323 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:53.279 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:07.325 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:53.271 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:07.329 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:53.285 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:07.329 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:53.275 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:07.356 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:35:53.301 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:07.358 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:35:53.314 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:17.356 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:03.312 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:17.404 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:03.360 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:17.405 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:03.350 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:17.412 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:03.358 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:17.435 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:03.380 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:17.441 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:03.397 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:17.457 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:03.413 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:17.458 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:03.415 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:17.484 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:03.440 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:22.313 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:08.269 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:22.318 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:08.274 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:22.338 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:08.294 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:22.366 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:08.311 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:22.370 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:08.326 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:22.372 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:08.317 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:22.374 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:08.330 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:22.390 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:08.346 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:22.393 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:08.338 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:37.377 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:23.333 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:37.380 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:23.336 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:37.407 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:23.363 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:37.422 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:23.367 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:37.425 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:23.370 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:37.444 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:23.400 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:37.447 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:23.403 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:37.454 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:23.399 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:37.533 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:23.489 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:38.261 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:24.206 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:38.262 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:24.207 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:38.271 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:24.217 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:38.302 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:24.258 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:38.303 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:24.259 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:38.315 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:24.271 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:38.322 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:24.279 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:38.324 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:24.280 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:38.334 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:24.289 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:47.726 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:33.682 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:47.781 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:33.737 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:47.813 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:33.769 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:47.848 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:33.793 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:47.853 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:33.798 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:47.856 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:33.812 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:47.860 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:33.816 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:47.891 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:33.847 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:47.892 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:33.837 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:52.422 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:38.378 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:52.425 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:38.381 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:52.440 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:38.396 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:52.469 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:38.414 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:52.473 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:38.418 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:52.473 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:38.430 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:52.477 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:38.433 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:52.486 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:38.432 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:52.491 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:38.447 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:57.525 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:43.481 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:57.528 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:43.484 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:57.554 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:43.510 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:57.570 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:43.516 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:57.575 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:43.521 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:57.575 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:43.532 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:57.592 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:43.537 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:57.600 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:43.556 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:18:57.621 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:43.577 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:07.433 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:53.389 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:07.481 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:53.437 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:07.512 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:36:53.467 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:07.527 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:53.472 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:07.530 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:53.476 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:07.532 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:53.488 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:07.536 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:53.492 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:07.551 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:36:53.496 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:07.559 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:36:53.515 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:17.515 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:03.471 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:17.518 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:03.474 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:17.543 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:03.499 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:17.562 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:03.507 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:17.565 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:03.521 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:17.565 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:03.511 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:17.569 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:03.525 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:17.589 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:03.534 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:17.594 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:03.550 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:22.638 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:08.594 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:22.646 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:08.602 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:22.664 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:08.620 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:22.694 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:08.639 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:22.705 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:08.650 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:22.705 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:08.662 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:22.708 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:08.664 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:22.720 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:08.666 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:22.723 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:08.680 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:27.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripAppendPP (3352 ms) 2024-01-27T04:19:27.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-01-27T04:19:27.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3544 ms) 2024-01-27T04:19:27.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-01-27T04:19:27.036 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3292 ms) 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3424 ms) 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: setting pool EIO 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: max_success 25, min_failed 26 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (5386 ms) 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3573 ms) 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (1940536 ms total) 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-01-27T04:19:27.037 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3409 ms) 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9391 ms) 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (17285 ms) 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3484 ms) 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (33569 ms total) 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3378 ms) 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3378 ms total) 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-01-27T04:19:27.038 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (15208 ms) 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7133 ms) 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7679 ms) 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7473 ms) 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3475 ms) 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7820 ms) 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (8047 ms) 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-01-27T04:19:27.039 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7148 ms) 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6680 ms) 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (8058 ms) 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7001 ms) 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7707 ms) 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-01-27T04:19:27.040 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6796 ms) 2024-01-27T04:19:37.682 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:23.638 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:37.685 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:23.641 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:37.712 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:23.668 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:37.729 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:23.675 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:37.731 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:23.687 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:37.734 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:23.689 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:37.735 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:23.680 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:37.764 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:23.709 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:37.765 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:23.721 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:52.635 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:38.590 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:52.638 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:38.594 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:52.670 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:38.626 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:52.685 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:38.631 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:52.687 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:38.632 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:52.689 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:38.645 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:52.690 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:38.646 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:52.714 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:38.659 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:52.720 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:38.676 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:57.811 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:43.767 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:57.865 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:43.821 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:57.884 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:43.840 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:57.916 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:43.872 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:57.916 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:43.862 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:57.918 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:43.863 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:57.935 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:43.880 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:57.938 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:43.894 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:19:57.960 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:43.916 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:12.790 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:58.745 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:12.792 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:58.748 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:12.837 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:37:58.793 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:12.839 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:58.785 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:12.841 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:58.786 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:12.866 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:58.822 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:12.869 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:37:58.814 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:12.875 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:58.830 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:12.903 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:37:58.859 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:16.719 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-01-27T04:20:16.719 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3409 ms) 2024-01-27T04:20:16.719 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-01-27T04:20:16.719 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7311 ms) 2024-01-27T04:20:16.719 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-01-27T04:20:16.719 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7289 ms) 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (8003 ms) 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (8061 ms) 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7810 ms) 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7797 ms) 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (149905 ms total) 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (2127389 ms total) 2024-01-27T04:20:16.720 INFO:tasks.workunit.client.0.smithi110.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-01-27T04:20:16.724 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.724 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40092 2024-01-27T04:20:16.724 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40092 2024-01-27T04:20:16.724 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.724 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41085 2024-01-27T04:20:16.724 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 41085 2024-01-27T04:20:16.724 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.724 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40374 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40374 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41187 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 41187 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40448 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40448 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40915 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40915 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41269 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 41269 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.725 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40413 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40413 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40795 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40795 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41064 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 41064 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40900 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40900 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41149 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 41149 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.726 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40725 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40725 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40290 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40290 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41222 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 41222 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40599 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40599 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40064 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40064 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.727 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40123 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40123 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40533 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40533 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40165 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40165 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40645 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40645 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=40680 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 40680 2024-01-27T04:20:16.728 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.729 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41025 2024-01-27T04:20:16.729 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 41025 2024-01-27T04:20:16.729 INFO:tasks.workunit.client.0.smithi110.stderr:+ for t in "${!pids[@]}" 2024-01-27T04:20:16.729 INFO:tasks.workunit.client.0.smithi110.stderr:+ pid=41277 2024-01-27T04:20:16.729 INFO:tasks.workunit.client.0.smithi110.stderr:+ wait 41277 2024-01-27T04:20:16.729 INFO:tasks.workunit.client.0.smithi110.stderr:+ exit 0 2024-01-27T04:20:16.729 INFO:tasks.workunit.client.0.smithi110.stderr:+ cleanup 2024-01-27T04:20:16.729 INFO:tasks.workunit.client.0.smithi110.stderr:+ pkill -P 40059 2024-01-27T04:20:16.834 INFO:tasks.workunit.client.0.smithi110.stderr:+ true 2024-01-27T04:20:16.835 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-01-27T04:20:16.835 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-01-27T04:20:16.870 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-01-27T04:20:16.870 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-01-27T04:20:17.169 DEBUG:teuthology.parallel:result is None 2024-01-27T04:20:17.169 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-01-27T04:20:17.196 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-01-27T04:20:17.196 DEBUG:teuthology.orchestra.run.smithi110:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-01-27T04:20:17.262 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-01-27T04:20:17.263 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-27T04:20:17.272 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-27T04:20:17.272 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T04:20:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.401+0000 7ff39aade640 10 monclient: get_monmap_and_config 2024-01-27T04:20:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.401+0000 7ff39aade640 10 monclient: build_initial_monmap 2024-01-27T04:20:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.401+0000 7ff39aade640 10 monclient: monmap: 2024-01-27T04:20:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:17.402 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:17.402584+0000 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:17.402584+0000 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:17.403 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:17.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 1 Processor -- start 2024-01-27T04:20:17.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 1 -- start start 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient: init 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff39406a6d0 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff39406aab0 0x7ff394145590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient(hunting): picked mon.noname-a con 0x7ff39406aab0 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff3940a9db0 0x7ff394145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient(hunting): picked mon.noname-c con 0x7ff3940a9db0 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3940b3ff0 con 0x7ff39406a6d0 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3940aae50 con 0x7ff39406aab0 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3940ab130 con 0x7ff3940a9db0 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:17.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.402+0000 7ff39aade640 10 monclient(hunting): authenticate will time out at 3700.756819s 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.403+0000 7ff393fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff39406aab0 0x7ff394145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.403+0000 7ff39907b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff3940a9db0 0x7ff394145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.403+0000 7ff39907b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff3940a9db0 0x7ff394145ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:56424/0 (socket says 172.21.15.110:56424) 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.403+0000 7ff39907b640 1 -- 172.21.15.110:0/2364341937 learned_addr learned my addr 172.21.15.110:0/2364341937 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.403+0000 7ff39907b640 10 monclient(hunting): get_auth_request con 0x7ff3940a9db0 auth_method 0 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.403+0000 7ff39907b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.403+0000 7ff39907b640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.403+0000 7ff39907b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.404+0000 7ff393fff640 10 monclient(hunting): get_auth_request con 0x7ff39406aab0 auth_method 0 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.404+0000 7ff393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.404+0000 7ff393fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.404+0000 7ff393fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.404+0000 7ff39907b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.404+0000 7ff39907b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:17.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.404+0000 7ff39907b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:17.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.405+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.405+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.405+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.406+0000 7ff39907b640 10 monclient(hunting): handle_auth_done global_id 20884 payload 274 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.406+0000 7ff39907b640 1 -- 172.21.15.110:0/2364341937 >> v2:172.21.15.110:3300/0 conn(0x7ff39406aab0 msgr2=0x7ff394145590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.406+0000 7ff39907b640 1 --2- 172.21.15.110:0/2364341937 >> v2:172.21.15.110:3300/0 conn(0x7ff39406aab0 0x7ff394145590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.406+0000 7ff39907b640 1 -- 172.21.15.110:0/2364341937 >> v1:172.21.15.110:6789/0 conn(0x7ff39406a6d0 legacy=0x7ff3940b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.406+0000 7ff39907b640 10 monclient: _finish_hunting 0 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.406+0000 7ff39907b640 1 monclient: found mon.noname-c 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.406+0000 7ff39907b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.406+0000 7ff39907b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.406+0000 7ff39907b640 1 -- 172.21.15.110:0/2364341937 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3940b4290 con 0x7ff3940a9db0 2024-01-27T04:20:17.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.407+0000 7ff39907b640 1 --2- 172.21.15.110:0/2364341937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3940a9db0 0x7ff394145ad0 secure :-1 s=READY pgs=5079 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009e50 tx=0x7ff38c02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92029f5a8c4fc0f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:17.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 1 -- 172.21.15.110:0/2364341937 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c02ca80 con 0x7ff3940a9db0 2024-01-27T04:20:17.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:17.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:17.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 10 monclient: dump: 2024-01-27T04:20:17.412 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 10 monclient: _finish_auth 0 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 10 monclient: _check_auth_tickets 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 1 -- 172.21.15.110:0/2364341937 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c02cbe0 con 0x7ff3940a9db0 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff39aade640 5 monclient: authenticate success, global_id 20884 2024-01-27T04:20:17.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff39aade640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff39aade640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff39aade640 4 set_mon_vals no callback set 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 1 -- 172.21.15.110:0/2364341937 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c0343f0 con 0x7ff3940a9db0 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.411+0000 7ff3937fe640 10 monclient: dump: 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:17.414 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:17.415 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:17.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 10 monclient: shutdown 2024-01-27T04:20:17.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:17.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 1 -- 172.21.15.110:0/2364341937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3940a9db0 msgr2=0x7ff394145ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:17.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 1 --2- 172.21.15.110:0/2364341937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3940a9db0 0x7ff394145ad0 secure :-1 s=READY pgs=5079 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009e50 tx=0x7ff38c02bee0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 1 -- 172.21.15.110:0/2364341937 shutdown_connections 2024-01-27T04:20:17.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 1 --2- 172.21.15.110:0/2364341937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3940a9db0 0x7ff394145ad0 unknown :-1 s=CLOSED pgs=5079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 1 --2- 172.21.15.110:0/2364341937 >> v2:172.21.15.110:3300/0 conn(0x7ff39406aab0 0x7ff394145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 1 -- 172.21.15.110:0/2364341937 >> 172.21.15.110:0/2364341937 conn(0x7ff3940b7e40 msgr2=0x7ff3940b8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 1 -- 172.21.15.110:0/2364341937 shutdown_connections 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 1 -- 172.21.15.110:0/2364341937 wait complete. 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 10 monclient: build_initial_monmap 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.412+0000 7ff39aade640 10 monclient: monmap: 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:17.417 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 librados: starting msgr at 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 librados: starting objecter 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 Processor -- start 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 -- start start 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 librados: setting wanted keys 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 librados: calling monclient init 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient: init 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:17.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39406a6d0 0x7ff3941420a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient(hunting): picked mon.noname-b con 0x7ff39406a6d0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3940a9db0 0x7ff39413ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient(hunting): picked mon.noname-a con 0x7ff3940a9db0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff39413a260 0x7ff39413c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient(hunting): picked mon.noname-c con 0x7ff39413a260 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39413eb00 con 0x7ff3940a9db0 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff394139170 con 0x7ff39406a6d0 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:17.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39413f4e0 con 0x7ff39413a260 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.413+0000 7ff39aade640 10 monclient(hunting): authenticate will time out at 3700.767922s 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff393fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3940a9db0 0x7ff39413ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39907b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff39413a260 0x7ff39413c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39887a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39406a6d0 0x7ff3941420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39907b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff39413a260 0x7ff39413c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:41066/0 (socket says 172.21.15.110:41066) 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff393fff640 10 monclient(hunting): get_auth_request con 0x7ff3940a9db0 auth_method 0 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39907b640 1 -- 172.21.15.110:0/1186446102 learned_addr learned my addr 172.21.15.110:0/1186446102 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff393fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff393fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39907b640 10 monclient(hunting): get_auth_request con 0x7ff39413a260 auth_method 0 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39907b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39907b640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:17.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39907b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39887a640 10 monclient(hunting): get_auth_request con 0x7ff39406a6d0 auth_method 0 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39887a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39887a640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.414+0000 7ff39887a640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.415+0000 7ff39887a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.415+0000 7ff39887a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.415+0000 7ff39887a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.415+0000 7ff39907b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.415+0000 7ff39907b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.415+0000 7ff39907b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.416+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.416+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.416+0000 7ff393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 10 monclient(hunting): handle_auth_done global_id 20887 payload 824 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 1 -- 172.21.15.110:0/1186446102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff39413a260 msgr2=0x7ff39413c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff39413a260 0x7ff39413c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 1 -- 172.21.15.110:0/1186446102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3940a9db0 msgr2=0x7ff39413ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:17.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3940a9db0 0x7ff39413ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 10 monclient: _finish_hunting 0 2024-01-27T04:20:17.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 1 monclient: found mon.noname-b 2024-01-27T04:20:17.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:17.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:17.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.418+0000 7ff39887a640 1 -- 172.21.15.110:0/1186446102 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c009b30 con 0x7ff39406a6d0 2024-01-27T04:20:17.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.420+0000 7ff39887a640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39406a6d0 0x7ff3941420a0 secure :-1 s=READY pgs=5080 cs=0 l=1 rev1=1 crypto rx=0x7ff38800c340 tx=0x7ff38800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:17.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.420+0000 7ff3917fa640 10 client.?.objecter ms_handle_connect 0x7ff39406a6d0 2024-01-27T04:20:17.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.420+0000 7ff3917fa640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:17.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff3917fa640 1 -- 172.21.15.110:0/1186446102 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3880052d0 con 0x7ff39406a6d0 2024-01-27T04:20:17.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff3917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff3917fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff3917fa640 10 monclient: dump: 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff3917fa640 10 monclient: _finish_auth 0 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff3917fa640 10 monclient: _check_auth_tickets 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff3917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:17.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff3917fa640 1 -- 172.21.15.110:0/1186446102 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff388005450 con 0x7ff39406a6d0 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff39aade640 5 monclient: authenticate success, global_id 20887 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff39aade640 10 monclient: _renew_subs 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff39aade640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.423+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff394139930 con 0x7ff39406a6d0 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff3917fa640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff3917fa640 1 -- 172.21.15.110:0/1186446102 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388007260 con 0x7ff39406a6d0 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff3917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff3917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff3917fa640 10 monclient: dump: 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:17.427 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:17.428 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:17.428 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:17.428 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:17.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff392ffd640 4 set_mon_vals no callback set 2024-01-27T04:20:17.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff39aade640 10 client.20887.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:17.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff39aade640 10 monclient: _renew_subs 2024-01-27T04:20:17.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff39aade640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:17.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39413e700 con 0x7ff39406a6d0 2024-01-27T04:20:17.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.424+0000 7ff39aade640 1 librados: init done 2024-01-27T04:20:17.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.425+0000 7ff39aade640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:17.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.425+0000 7ff39aade640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:17.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.425+0000 7ff39aade640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:17.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.425+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39404ea50 con 0x7ff39406a6d0 2024-01-27T04:20:17.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.674+0000 7ff3917fa640 1 -- 172.21.15.110:0/1186446102 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7ff3880073c0 con 0x7ff39406a6d0 2024-01-27T04:20:17.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.674+0000 7ff3917fa640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff37806c9f0 0x7ff37806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:17.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.674+0000 7ff393fff640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff37806c9f0 0x7ff37806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:17.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.674+0000 7ff393fff640 10 monclient: get_auth_request con 0x7ff37806c9f0 auth_method 0 2024-01-27T04:20:17.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.675+0000 7ff393fff640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff37806c9f0 0x7ff37806eeb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff3941428f0 tx=0x7ff380009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:17.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.675+0000 7ff3917fa640 10 client.20887.objecter ms_handle_connect 0x7ff37806c9f0 2024-01-27T04:20:17.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.676+0000 7ff3917fa640 1 -- 172.21.15.110:0/1186446102 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7ff388057bf0 con 0x7ff39406a6d0 2024-01-27T04:20:17.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.676+0000 7ff3917fa640 10 client.20887.objecter ms_dispatch 0x7ff394000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:17.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.676+0000 7ff3917fa640 3 client.20887.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:17.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.676+0000 7ff3917fa640 3 client.20887.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:17.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.676+0000 7ff3917fa640 20 client.20887.objecter dump_active .. 0 homeless 2024-01-27T04:20:17.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.687+0000 7ff3917fa640 1 -- 172.21.15.110:0/1186446102 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7ff388053e90 con 0x7ff39406a6d0 2024-01-27T04:20:17.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.687+0000 7ff3917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:17.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.687+0000 7ff3917fa640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:17.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.866+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff39414c890 con 0x7ff37806c9f0 2024-01-27T04:20:17.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.871+0000 7ff3917fa640 1 -- 172.21.15.110:0/1186446102 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151868 (secure 0 0 0) 0x7ff380011a30 con 0x7ff37806c9f0 2024-01-27T04:20:17.874 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T04:20:17.875 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.873+0000 7ff39aade640 10 librados: watch_flush enter 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 10 librados: watch_flush exit 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 20 client.20887.objecter shutdown clearing up homeless session... 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 10 client.20887.objecter successfully canceled tick 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff37806c9f0 msgr2=0x7ff37806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff37806c9f0 0x7ff37806eeb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff3941428f0 tx=0x7ff380009190 comp rx=0 tx=0).stop 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 10 monclient: shutdown 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39406a6d0 msgr2=0x7ff3941420a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39406a6d0 0x7ff3941420a0 secure :-1 s=READY pgs=5080 cs=0 l=1 rev1=1 crypto rx=0x7ff38800c340 tx=0x7ff38800c800 comp rx=0 tx=0).stop 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 shutdown_connections 2024-01-27T04:20:17.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff37806c9f0 0x7ff37806eeb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff39413a260 0x7ff39413c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3940a9db0 0x7ff39413ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 --2- 172.21.15.110:0/1186446102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39406a6d0 0x7ff3941420a0 unknown :-1 s=CLOSED pgs=5080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 >> 172.21.15.110:0/1186446102 conn(0x7ff3940b7e40 msgr2=0x7ff3940b4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 shutdown_connections 2024-01-27T04:20:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 -- 172.21.15.110:0/1186446102 wait complete. 2024-01-27T04:20:17.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.874+0000 7ff39aade640 1 librados: shutdown 2024-01-27T04:20:17.883 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:03.839 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:17.886 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":2451,"stamp":"2024-01-27T04:20:17.553087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":578036,"kb_used_data":39808,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754396684,"statfs":{"total":773094113280,"available":772502204416,"internally_reserved":0,"allocated":40763392,"data_stored":19076953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":151,"apply_latency_ms":151,"commit_latency_ns":151000000,"apply_latency_ns":151000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006263"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2557,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.713917+0000","last_change":"2024-01-27T03:45:12.104231+0000","last_active":"2024-01-27T04:20:16.713917+0000","last_peered":"2024-01-27T04:20:16.713917+0000","last_clean":"2024-01-27T04:20:16.713917+0000","last_became_active":"2024-01-27T03:45:12.095646+0000","last_became_peered":"2024-01-27T03:45:12.095646+0000","last_unstale":"2024-01-27T04:20:16.713917+0000","last_undegraded":"2024-01-27T04:20:16.713917+0000","last_fullsized":"2024-01-27T04:20:16.713917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:14:48.806131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2561,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.335104+0000","last_change":"2024-01-27T03:45:50.347781+0000","last_active":"2024-01-27T04:20:12.335104+0000","last_peered":"2024-01-27T04:20:12.335104+0000","last_clean":"2024-01-27T04:20:12.335104+0000","last_became_active":"2024-01-27T03:45:50.339311+0000","last_became_peered":"2024-01-27T03:45:50.339311+0000","last_unstale":"2024-01-27T04:20:12.335104+0000","last_undegraded":"2024-01-27T04:20:12.335104+0000","last_fullsized":"2024-01-27T04:20:12.335104+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:32.762753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2560,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.761038+0000","last_change":"2024-01-27T03:45:01.242765+0000","last_active":"2024-01-27T04:20:15.761038+0000","last_peered":"2024-01-27T04:20:15.761038+0000","last_clean":"2024-01-27T04:20:15.761038+0000","last_became_active":"2024-01-27T03:45:01.173317+0000","last_became_peered":"2024-01-27T03:45:01.173317+0000","last_unstale":"2024-01-27T04:20:15.761038+0000","last_undegraded":"2024-01-27T04:20:15.761038+0000","last_fullsized":"2024-01-27T04:20:15.761038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:45:07.062762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2547,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.781859+0000","last_change":"2024-01-27T03:45:08.326818+0000","last_active":"2024-01-27T04:20:13.781859+0000","last_peered":"2024-01-27T04:20:13.781859+0000","last_clean":"2024-01-27T04:20:13.781859+0000","last_became_active":"2024-01-27T03:45:08.187722+0000","last_became_peered":"2024-01-27T03:45:08.187722+0000","last_unstale":"2024-01-27T04:20:13.781859+0000","last_undegraded":"2024-01-27T04:20:13.781859+0000","last_fullsized":"2024-01-27T04:20:13.781859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:07:16.791137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2548,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.875693+0000","last_change":"2024-01-27T03:45:03.762871+0000","last_active":"2024-01-27T04:20:13.875693+0000","last_peered":"2024-01-27T04:20:13.875693+0000","last_clean":"2024-01-27T04:20:13.875693+0000","last_became_active":"2024-01-27T03:45:03.757221+0000","last_became_peered":"2024-01-27T03:45:03.757221+0000","last_unstale":"2024-01-27T04:20:13.875693+0000","last_undegraded":"2024-01-27T04:20:13.875693+0000","last_fullsized":"2024-01-27T04:20:13.875693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:52.156074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2572,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.736771+0000","last_change":"2024-01-27T03:45:01.929664+0000","last_active":"2024-01-27T04:20:14.736771+0000","last_peered":"2024-01-27T04:20:14.736771+0000","last_clean":"2024-01-27T04:20:14.736771+0000","last_became_active":"2024-01-27T03:45:01.916892+0000","last_became_peered":"2024-01-27T03:45:01.916892+0000","last_unstale":"2024-01-27T04:20:14.736771+0000","last_undegraded":"2024-01-27T04:20:14.736771+0000","last_fullsized":"2024-01-27T04:20:14.736771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:02:49.278494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2561,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.352601+0000","last_change":"2024-01-27T03:45:50.416152+0000","last_active":"2024-01-27T04:20:12.352601+0000","last_peered":"2024-01-27T04:20:12.352601+0000","last_clean":"2024-01-27T04:20:12.352601+0000","last_became_active":"2024-01-27T03:45:50.401687+0000","last_became_peered":"2024-01-27T03:45:50.401687+0000","last_unstale":"2024-01-27T04:20:12.352601+0000","last_undegraded":"2024-01-27T04:20:12.352601+0000","last_fullsized":"2024-01-27T04:20:12.352601+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:34:59.292181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2563,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.357036+0000","last_change":"2024-01-27T03:45:16.267960+0000","last_active":"2024-01-27T04:20:12.357036+0000","last_peered":"2024-01-27T04:20:12.357036+0000","last_clean":"2024-01-27T04:20:12.357036+0000","last_became_active":"2024-01-27T03:45:16.039935+0000","last_became_peered":"2024-01-27T03:45:16.039935+0000","last_unstale":"2024-01-27T04:20:12.357036+0000","last_undegraded":"2024-01-27T04:20:12.357036+0000","last_fullsized":"2024-01-27T04:20:12.357036+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:02:05.330053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"48'7","reported_seq":2588,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.717267+0000","last_change":"2024-01-27T03:45:00.305112+0000","last_active":"2024-01-27T04:20:14.717267+0000","last_peered":"2024-01-27T04:20:14.717267+0000","last_clean":"2024-01-27T04:20:14.717267+0000","last_became_active":"2024-01-27T03:45:00.291987+0000","last_became_peered":"2024-01-27T03:45:00.291987+0000","last_unstale":"2024-01-27T04:20:14.717267+0000","last_undegraded":"2024-01-27T04:20:14.717267+0000","last_fullsized":"2024-01-27T04:20:14.717267+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:37.876016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2550,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.745980+0000","last_change":"2024-01-27T03:44:59.103125+0000","last_active":"2024-01-27T04:20:13.745980+0000","last_peered":"2024-01-27T04:20:13.745980+0000","last_clean":"2024-01-27T04:20:13.745980+0000","last_became_active":"2024-01-27T03:44:59.080371+0000","last_became_peered":"2024-01-27T03:44:59.080371+0000","last_unstale":"2024-01-27T04:20:13.745980+0000","last_undegraded":"2024-01-27T04:20:13.745980+0000","last_fullsized":"2024-01-27T04:20:13.745980+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:17.291036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2538,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.585376+0000","last_change":"2024-01-27T03:45:05.767658+0000","last_active":"2024-01-27T04:20:15.585376+0000","last_peered":"2024-01-27T04:20:15.585376+0000","last_clean":"2024-01-27T04:20:15.585376+0000","last_became_active":"2024-01-27T03:45:05.758355+0000","last_became_peered":"2024-01-27T03:45:05.758355+0000","last_unstale":"2024-01-27T04:20:15.585376+0000","last_undegraded":"2024-01-27T04:20:15.585376+0000","last_fullsized":"2024-01-27T04:20:15.585376+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:29:16.853701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2555,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.721456+0000","last_change":"2024-01-27T03:45:11.894875+0000","last_active":"2024-01-27T04:20:16.721456+0000","last_peered":"2024-01-27T04:20:16.721456+0000","last_clean":"2024-01-27T04:20:16.721456+0000","last_became_active":"2024-01-27T03:45:11.884947+0000","last_became_peered":"2024-01-27T03:45:11.884947+0000","last_unstale":"2024-01-27T04:20:16.721456+0000","last_undegraded":"2024-01-27T04:20:16.721456+0000","last_fullsized":"2024-01-27T04:20:16.721456+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:23:21.445636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":2577,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.279707+0000","last_change":"2024-01-27T03:43:25.035922+0000","last_active":"2024-01-27T04:20:12.279707+0000","last_peered":"2024-01-27T04:20:12.279707+0000","last_clean":"2024-01-27T04:20:12.279707+0000","last_became_active":"2024-01-27T03:43:24.404196+0000","last_became_peered":"2024-01-27T03:43:24.404196+0000","last_unstale":"2024-01-27T04:20:12.279707+0000","last_undegraded":"2024-01-27T04:20:12.279707+0000","last_fullsized":"2024-01-27T04:20:12.279707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_clean_scrub_stamp":"2024-01-27T03:43:13.902947+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:56:03.692684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2581,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.460782+0000","last_change":"2024-01-27T03:43:30.313647+0000","last_active":"2024-01-27T04:20:16.460782+0000","last_peered":"2024-01-27T04:20:16.460782+0000","last_clean":"2024-01-27T04:20:16.460782+0000","last_became_active":"2024-01-27T03:43:30.303196+0000","last_became_peered":"2024-01-27T03:43:30.303196+0000","last_unstale":"2024-01-27T04:20:16.460782+0000","last_undegraded":"2024-01-27T04:20:16.460782+0000","last_fullsized":"2024-01-27T04:20:16.460782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:45:18.950474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2549,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.750175+0000","last_change":"2024-01-27T03:45:06.207200+0000","last_active":"2024-01-27T04:20:13.750175+0000","last_peered":"2024-01-27T04:20:13.750175+0000","last_clean":"2024-01-27T04:20:13.750175+0000","last_became_active":"2024-01-27T03:45:06.197607+0000","last_became_peered":"2024-01-27T03:45:06.197607+0000","last_unstale":"2024-01-27T04:20:13.750175+0000","last_undegraded":"2024-01-27T04:20:13.750175+0000","last_fullsized":"2024-01-27T04:20:13.750175+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:30:59.795065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2583,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.446781+0000","last_change":"2024-01-27T03:43:30.123281+0000","last_active":"2024-01-27T04:20:16.446781+0000","last_peered":"2024-01-27T04:20:16.446781+0000","last_clean":"2024-01-27T04:20:16.446781+0000","last_became_active":"2024-01-27T03:43:30.033663+0000","last_became_peered":"2024-01-27T03:43:30.033663+0000","last_unstale":"2024-01-27T04:20:16.446781+0000","last_undegraded":"2024-01-27T04:20:16.446781+0000","last_fullsized":"2024-01-27T04:20:16.446781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:03.978730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2548,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.773182+0000","last_change":"2024-01-27T03:45:07.001823+0000","last_active":"2024-01-27T04:20:13.773182+0000","last_peered":"2024-01-27T04:20:13.773182+0000","last_clean":"2024-01-27T04:20:13.773182+0000","last_became_active":"2024-01-27T03:45:06.986316+0000","last_became_peered":"2024-01-27T03:45:06.986316+0000","last_unstale":"2024-01-27T04:20:13.773182+0000","last_undegraded":"2024-01-27T04:20:13.773182+0000","last_fullsized":"2024-01-27T04:20:13.773182+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:42.117379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2584,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.718217+0000","last_change":"2024-01-27T03:43:30.100170+0000","last_active":"2024-01-27T04:20:14.718217+0000","last_peered":"2024-01-27T04:20:14.718217+0000","last_clean":"2024-01-27T04:20:14.718217+0000","last_became_active":"2024-01-27T03:43:30.072010+0000","last_became_peered":"2024-01-27T03:43:30.072010+0000","last_unstale":"2024-01-27T04:20:14.718217+0000","last_undegraded":"2024-01-27T04:20:14.718217+0000","last_fullsized":"2024-01-27T04:20:14.718217+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:17:46.003541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2553,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.294627+0000","last_change":"2024-01-27T03:45:28.248666+0000","last_active":"2024-01-27T04:20:12.294627+0000","last_peered":"2024-01-27T04:20:12.294627+0000","last_clean":"2024-01-27T04:20:12.294627+0000","last_became_active":"2024-01-27T03:45:28.239949+0000","last_became_peered":"2024-01-27T03:45:28.239949+0000","last_unstale":"2024-01-27T04:20:12.294627+0000","last_undegraded":"2024-01-27T04:20:12.294627+0000","last_fullsized":"2024-01-27T04:20:12.294627+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:11:52.211119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.458866+0000","last_change":"2024-01-27T03:43:30.335032+0000","last_active":"2024-01-27T04:20:16.458866+0000","last_peered":"2024-01-27T04:20:16.458866+0000","last_clean":"2024-01-27T04:20:16.458866+0000","last_became_active":"2024-01-27T03:43:30.329967+0000","last_became_peered":"2024-01-27T03:43:30.329967+0000","last_unstale":"2024-01-27T04:20:16.458866+0000","last_undegraded":"2024-01-27T04:20:16.458866+0000","last_fullsized":"2024-01-27T04:20:16.458866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:30.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2549,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.741986+0000","last_change":"2024-01-27T03:45:03.469717+0000","last_active":"2024-01-27T04:20:13.741986+0000","last_peered":"2024-01-27T04:20:13.741986+0000","last_clean":"2024-01-27T04:20:13.741986+0000","last_became_active":"2024-01-27T03:45:03.435938+0000","last_became_peered":"2024-01-27T03:45:03.435938+0000","last_unstale":"2024-01-27T04:20:13.741986+0000","last_undegraded":"2024-01-27T04:20:13.741986+0000","last_fullsized":"2024-01-27T04:20:13.741986+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:31:47.895364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2559,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.872098+0000","last_change":"2024-01-27T03:43:30.340472+0000","last_active":"2024-01-27T04:20:13.872098+0000","last_peered":"2024-01-27T04:20:13.872098+0000","last_clean":"2024-01-27T04:20:13.872098+0000","last_became_active":"2024-01-27T03:43:30.329560+0000","last_became_peered":"2024-01-27T03:43:30.329560+0000","last_unstale":"2024-01-27T04:20:13.872098+0000","last_undegraded":"2024-01-27T04:20:13.872098+0000","last_fullsized":"2024-01-27T04:20:13.872098+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:10:58.217001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2560,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.350888+0000","last_change":"2024-01-27T03:45:11.682176+0000","last_active":"2024-01-27T04:20:12.350888+0000","last_peered":"2024-01-27T04:20:12.350888+0000","last_clean":"2024-01-27T04:20:12.350888+0000","last_became_active":"2024-01-27T03:45:11.344546+0000","last_became_peered":"2024-01-27T03:45:11.344546+0000","last_unstale":"2024-01-27T04:20:12.350888+0000","last_undegraded":"2024-01-27T04:20:12.350888+0000","last_fullsized":"2024-01-27T04:20:12.350888+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:35:01.350866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2581,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.473814+0000","last_change":"2024-01-27T03:43:30.077908+0000","last_active":"2024-01-27T04:20:16.473814+0000","last_peered":"2024-01-27T04:20:16.473814+0000","last_clean":"2024-01-27T04:20:16.473814+0000","last_became_active":"2024-01-27T03:43:30.057792+0000","last_became_peered":"2024-01-27T03:43:30.057792+0000","last_unstale":"2024-01-27T04:20:16.473814+0000","last_undegraded":"2024-01-27T04:20:16.473814+0000","last_fullsized":"2024-01-27T04:20:16.473814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:05.446156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2562,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.345492+0000","last_change":"2024-01-27T03:45:12.756405+0000","last_active":"2024-01-27T04:20:12.345492+0000","last_peered":"2024-01-27T04:20:12.345492+0000","last_clean":"2024-01-27T04:20:12.345492+0000","last_became_active":"2024-01-27T03:45:12.751200+0000","last_became_peered":"2024-01-27T03:45:12.751200+0000","last_unstale":"2024-01-27T04:20:12.345492+0000","last_undegraded":"2024-01-27T04:20:12.345492+0000","last_fullsized":"2024-01-27T04:20:12.345492+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:07:39.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2571,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.756811+0000","last_change":"2024-01-27T03:43:30.372081+0000","last_active":"2024-01-27T04:20:15.756811+0000","last_peered":"2024-01-27T04:20:15.756811+0000","last_clean":"2024-01-27T04:20:15.756811+0000","last_became_active":"2024-01-27T03:43:30.335129+0000","last_became_peered":"2024-01-27T03:43:30.335129+0000","last_unstale":"2024-01-27T04:20:15.756811+0000","last_undegraded":"2024-01-27T04:20:15.756811+0000","last_fullsized":"2024-01-27T04:20:15.756811+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:01.568710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2570,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.472197+0000","last_change":"2024-01-27T03:45:11.743478+0000","last_active":"2024-01-27T04:20:16.472197+0000","last_peered":"2024-01-27T04:20:16.472197+0000","last_clean":"2024-01-27T04:20:16.472197+0000","last_became_active":"2024-01-27T03:45:11.734547+0000","last_became_peered":"2024-01-27T03:45:11.734547+0000","last_unstale":"2024-01-27T04:20:16.472197+0000","last_undegraded":"2024-01-27T04:20:16.472197+0000","last_fullsized":"2024-01-27T04:20:16.472197+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:58.163231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2579,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.468607+0000","last_change":"2024-01-27T03:43:30.350497+0000","last_active":"2024-01-27T04:20:16.468607+0000","last_peered":"2024-01-27T04:20:16.468607+0000","last_clean":"2024-01-27T04:20:16.468607+0000","last_became_active":"2024-01-27T03:43:30.345375+0000","last_became_peered":"2024-01-27T03:43:30.345375+0000","last_unstale":"2024-01-27T04:20:16.468607+0000","last_undegraded":"2024-01-27T04:20:16.468607+0000","last_fullsized":"2024-01-27T04:20:16.468607+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:58.675410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2570,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.735085+0000","last_change":"2024-01-27T03:45:08.095965+0000","last_active":"2024-01-27T04:20:14.735085+0000","last_peered":"2024-01-27T04:20:14.735085+0000","last_clean":"2024-01-27T04:20:14.735085+0000","last_became_active":"2024-01-27T03:45:08.074261+0000","last_became_peered":"2024-01-27T03:45:08.074261+0000","last_unstale":"2024-01-27T04:20:14.735085+0000","last_undegraded":"2024-01-27T04:20:14.735085+0000","last_fullsized":"2024-01-27T04:20:14.735085+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T15:20:03.504139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2560,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.733166+0000","last_change":"2024-01-27T03:45:29.625526+0000","last_active":"2024-01-27T04:20:16.733166+0000","last_peered":"2024-01-27T04:20:16.733166+0000","last_clean":"2024-01-27T04:20:16.733166+0000","last_became_active":"2024-01-27T03:45:29.620303+0000","last_became_peered":"2024-01-27T03:45:29.620303+0000","last_unstale":"2024-01-27T04:20:16.733166+0000","last_undegraded":"2024-01-27T04:20:16.733166+0000","last_fullsized":"2024-01-27T04:20:16.733166+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:09.679551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2554,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.737195+0000","last_change":"2024-01-27T03:45:11.908907+0000","last_active":"2024-01-27T04:20:16.737195+0000","last_peered":"2024-01-27T04:20:16.737195+0000","last_clean":"2024-01-27T04:20:16.737195+0000","last_became_active":"2024-01-27T03:45:11.902875+0000","last_became_peered":"2024-01-27T03:45:11.902875+0000","last_unstale":"2024-01-27T04:20:16.737195+0000","last_undegraded":"2024-01-27T04:20:16.737195+0000","last_fullsized":"2024-01-27T04:20:16.737195+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T14:33:16.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2547,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.831239+0000","last_change":"2024-01-27T03:45:41.083249+0000","last_active":"2024-01-27T04:20:13.831239+0000","last_peered":"2024-01-27T04:20:13.831239+0000","last_clean":"2024-01-27T04:20:13.831239+0000","last_became_active":"2024-01-27T03:45:41.077954+0000","last_became_peered":"2024-01-27T03:45:41.077954+0000","last_unstale":"2024-01-27T04:20:13.831239+0000","last_undegraded":"2024-01-27T04:20:13.831239+0000","last_fullsized":"2024-01-27T04:20:13.831239+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:53:19.777797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2555,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.730962+0000","last_change":"2024-01-27T03:45:13.508628+0000","last_active":"2024-01-27T04:20:16.730962+0000","last_peered":"2024-01-27T04:20:16.730962+0000","last_clean":"2024-01-27T04:20:16.730962+0000","last_became_active":"2024-01-27T03:45:13.503336+0000","last_became_peered":"2024-01-27T03:45:13.503336+0000","last_unstale":"2024-01-27T04:20:16.730962+0000","last_undegraded":"2024-01-27T04:20:16.730962+0000","last_fullsized":"2024-01-27T04:20:16.730962+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:08:42.711117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2551,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.291640+0000","last_change":"2024-01-27T03:45:06.421706+0000","last_active":"2024-01-27T04:20:12.291640+0000","last_peered":"2024-01-27T04:20:12.291640+0000","last_clean":"2024-01-27T04:20:12.291640+0000","last_became_active":"2024-01-27T03:45:06.412514+0000","last_became_peered":"2024-01-27T03:45:06.412514+0000","last_unstale":"2024-01-27T04:20:12.291640+0000","last_undegraded":"2024-01-27T04:20:12.291640+0000","last_fullsized":"2024-01-27T04:20:12.291640+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:23.739645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2562,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.337984+0000","last_change":"2024-01-27T03:45:05.911725+0000","last_active":"2024-01-27T04:20:12.337984+0000","last_peered":"2024-01-27T04:20:12.337984+0000","last_clean":"2024-01-27T04:20:12.337984+0000","last_became_active":"2024-01-27T03:45:05.905013+0000","last_became_peered":"2024-01-27T03:45:05.905013+0000","last_unstale":"2024-01-27T04:20:12.337984+0000","last_undegraded":"2024-01-27T04:20:12.337984+0000","last_fullsized":"2024-01-27T04:20:12.337984+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:23:34.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2552,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.277864+0000","last_change":"2024-01-27T03:45:09.267062+0000","last_active":"2024-01-27T04:20:12.277864+0000","last_peered":"2024-01-27T04:20:12.277864+0000","last_clean":"2024-01-27T04:20:12.277864+0000","last_became_active":"2024-01-27T03:45:08.513799+0000","last_became_peered":"2024-01-27T03:45:08.513799+0000","last_unstale":"2024-01-27T04:20:12.277864+0000","last_undegraded":"2024-01-27T04:20:12.277864+0000","last_fullsized":"2024-01-27T04:20:12.277864+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:45.278278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2571,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.448883+0000","last_change":"2024-01-27T03:45:12.810545+0000","last_active":"2024-01-27T04:20:16.448883+0000","last_peered":"2024-01-27T04:20:16.448883+0000","last_clean":"2024-01-27T04:20:16.448883+0000","last_became_active":"2024-01-27T03:45:12.797603+0000","last_became_peered":"2024-01-27T03:45:12.797603+0000","last_unstale":"2024-01-27T04:20:16.448883+0000","last_undegraded":"2024-01-27T04:20:16.448883+0000","last_fullsized":"2024-01-27T04:20:16.448883+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:06:00.673252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2532,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.622160+0000","last_change":"2024-01-27T03:45:08.152790+0000","last_active":"2024-01-27T04:20:15.622160+0000","last_peered":"2024-01-27T04:20:15.622160+0000","last_clean":"2024-01-27T04:20:15.622160+0000","last_became_active":"2024-01-27T03:45:08.146505+0000","last_became_peered":"2024-01-27T03:45:08.146505+0000","last_unstale":"2024-01-27T04:20:15.622160+0000","last_undegraded":"2024-01-27T04:20:15.622160+0000","last_fullsized":"2024-01-27T04:20:15.622160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:50.587416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2559,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.349162+0000","last_change":"2024-01-27T03:45:10.396512+0000","last_active":"2024-01-27T04:20:12.349162+0000","last_peered":"2024-01-27T04:20:12.349162+0000","last_clean":"2024-01-27T04:20:12.349162+0000","last_became_active":"2024-01-27T03:45:10.387717+0000","last_became_peered":"2024-01-27T03:45:10.387717+0000","last_unstale":"2024-01-27T04:20:12.349162+0000","last_undegraded":"2024-01-27T04:20:12.349162+0000","last_fullsized":"2024-01-27T04:20:12.349162+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:12:16.534326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2548,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.785854+0000","last_change":"2024-01-27T03:45:41.124551+0000","last_active":"2024-01-27T04:20:13.785854+0000","last_peered":"2024-01-27T04:20:13.785854+0000","last_clean":"2024-01-27T04:20:13.785854+0000","last_became_active":"2024-01-27T03:45:41.119306+0000","last_became_peered":"2024-01-27T03:45:41.119306+0000","last_unstale":"2024-01-27T04:20:13.785854+0000","last_undegraded":"2024-01-27T04:20:13.785854+0000","last_fullsized":"2024-01-27T04:20:13.785854+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:52:56.858068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2569,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.470531+0000","last_change":"2024-01-27T03:45:15.265604+0000","last_active":"2024-01-27T04:20:16.470531+0000","last_peered":"2024-01-27T04:20:16.470531+0000","last_clean":"2024-01-27T04:20:16.470531+0000","last_became_active":"2024-01-27T03:45:15.259193+0000","last_became_peered":"2024-01-27T03:45:15.259193+0000","last_unstale":"2024-01-27T04:20:16.470531+0000","last_undegraded":"2024-01-27T04:20:16.470531+0000","last_fullsized":"2024-01-27T04:20:16.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:48.948534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313662,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68372,"kb_used_data":5248,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303468,"statfs":{"total":96636764160,"available":96566751232,"internally_reserved":0,"allocated":5373952,"data_stored":2671367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":11.641999999999999,"5min":22.148,"15min":18.57},"min":{"1min":4.3710000000000004,"5min":4.3710000000000004,"15min":4.3710000000000004},"max":{"1min":35.859000000000002,"5min":232.63900000000001,"15min":232.63900000000001},"last":32.728999999999999},{"interface":"front","average":{"1min":15.961,"5min":23.016999999999999,"15min":19.32},"min":{"1min":4.8449999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"max":{"1min":48.558999999999997,"5min":265.815,"15min":265.815},"last":29.667999999999999}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.598000000000001,"5min":27.135000000000002,"15min":23.359999999999999},"min":{"1min":6.415,"5min":6.415,"15min":5.4749999999999996},"max":{"1min":55.116,"5min":152.09700000000001,"15min":421.90300000000002},"last":25.469999999999999},{"interface":"front","average":{"1min":12.83,"5min":26.446999999999999,"15min":21.591000000000001},"min":{"1min":6.3529999999999998,"5min":5.968,"15min":5.7039999999999997},"max":{"1min":40.463999999999999,"5min":165.53999999999999,"15min":319.13200000000001},"last":22.477}]},{"osd":3,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":16.913,"5min":26.349,"15min":24.771000000000001},"min":{"1min":7.9109999999999996,"5min":7.2750000000000004,"15min":6.0670000000000002},"max":{"1min":54.639000000000003,"5min":176.215,"15min":434.72800000000001},"last":37.106000000000002},{"interface":"front","average":{"1min":23.969000000000001,"5min":34.192999999999998,"15min":31.582999999999998},"min":{"1min":10.372999999999999,"5min":10.372999999999999,"15min":8.3019999999999996},"max":{"1min":95.001000000000005,"5min":216.80099999999999,"15min":447.06700000000001},"last":58.091999999999999}]},{"osd":4,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":19.324000000000002,"5min":24.849,"15min":23.100999999999999},"min":{"1min":8.2040000000000006,"5min":7.4240000000000004,"15min":7.0979999999999999},"max":{"1min":59.201000000000001,"5min":152.88,"15min":329.11500000000001},"last":38.07},{"interface":"front","average":{"1min":18.565999999999999,"5min":24.952999999999999,"15min":21.696999999999999},"min":{"1min":8.4870000000000001,"5min":7.1239999999999997,"15min":5.7839999999999998},"max":{"1min":61.326999999999998,"5min":233.845,"15min":233.845},"last":30.361999999999998}]},{"osd":5,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":25.951000000000001,"5min":30.747,"15min":26.004000000000001},"min":{"1min":8.7680000000000007,"5min":8.7680000000000007,"15min":7.165},"max":{"1min":101.246,"5min":252.73500000000001,"15min":324.52300000000002},"last":53.686},{"interface":"front","average":{"1min":27.847999999999999,"5min":38.902999999999999,"15min":30.933},"min":{"1min":13.746,"5min":11.076000000000001,"15min":9.0210000000000008},"max":{"1min":82.462999999999994,"5min":382,"15min":382},"last":35.701000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":29.972999999999999,"15min":28.385000000000002},"min":{"1min":7.335,"5min":7.335,"15min":7.335},"max":{"1min":33.107999999999997,"5min":165.976,"15min":400.93900000000002},"last":31.201000000000001},{"interface":"front","average":{"1min":16.268000000000001,"5min":28.919,"15min":27.251000000000001},"min":{"1min":9.4049999999999994,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":26.937999999999999,"5min":217.77699999999999,"15min":684.36699999999996},"last":34.392000000000003}]},{"osd":7,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":20.309999999999999,"5min":33.465000000000003,"15min":31.215},"min":{"1min":9.0519999999999996,"5min":8.1389999999999993,"15min":8.1389999999999993},"max":{"1min":50.585000000000001,"5min":242.08199999999999,"15min":443.21499999999997},"last":39.984000000000002},{"interface":"front","average":{"1min":27.695,"5min":32.735999999999997,"15min":28.863},"min":{"1min":12.496,"5min":9.6590000000000007,"15min":8.3309999999999995},"max":{"1min":133.63200000000001,"5min":145.751,"15min":436.80399999999997},"last":33.523000000000003}]}]},{"osd":2,"up_from":21,"seq":90194313660,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76292,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295548,"statfs":{"total":96636764160,"available":96558641152,"internally_reserved":0,"allocated":4972544,"data_stored":2273527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":8.7110000000000003,"5min":25.661999999999999,"15min":18.187999999999999},"min":{"1min":4.2939999999999996,"5min":4.2939999999999996,"15min":4.2939999999999996},"max":{"1min":23.433,"5min":445.62400000000002,"15min":445.62400000000002},"last":16.015999999999998},{"interface":"front","average":{"1min":12.178000000000001,"5min":34.600999999999999,"15min":22.532},"min":{"1min":7.1420000000000003,"5min":6.4370000000000003,"15min":5.3230000000000004},"max":{"1min":16.050000000000001,"5min":974.43299999999999,"15min":974.43299999999999},"last":21.056999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.199999999999999,"5min":37.404000000000003,"15min":25.739000000000001},"min":{"1min":7.0250000000000004,"5min":7.0250000000000004,"15min":6.6239999999999997},"max":{"1min":19.93,"5min":975.43499999999995,"15min":975.43499999999995},"last":8.5619999999999994},{"interface":"front","average":{"1min":12.119999999999999,"5min":28.672000000000001,"15min":22.556000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":5.5899999999999999},"max":{"1min":25.190000000000001,"5min":481.185,"15min":481.185},"last":12.433999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":17.236999999999998,"5min":36.584000000000003,"15min":26.488},"min":{"1min":7.665,"5min":7.2709999999999999,"15min":7.2709999999999999},"max":{"1min":59.902999999999999,"5min":980.83500000000004,"15min":980.83500000000004},"last":11.738},{"interface":"front","average":{"1min":18.442,"5min":41.698,"15min":29.129999999999999},"min":{"1min":11.558999999999999,"5min":10.58,"15min":9.5459999999999994},"max":{"1min":26.449000000000002,"5min":980.17499999999995,"15min":980.17499999999995},"last":15.047000000000001}]},{"osd":4,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":12.173999999999999,"5min":27.407,"15min":22.308},"min":{"1min":6.7809999999999997,"5min":6.2750000000000004,"15min":5.819},"max":{"1min":23.806000000000001,"5min":508.601,"15min":508.601},"last":8.2010000000000005},{"interface":"front","average":{"1min":14.736000000000001,"5min":34.642000000000003,"15min":27.102},"min":{"1min":9.7059999999999995,"5min":8.234,"15min":6.6580000000000004},"max":{"1min":27.920999999999999,"5min":492.29199999999997,"15min":492.29199999999997},"last":7.8460000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":18.317,"5min":48.280000000000001,"15min":33.753999999999998},"min":{"1min":8.9800000000000004,"5min":8.9800000000000004,"15min":8.9740000000000002},"max":{"1min":26.623999999999999,"5min":976.35900000000004,"15min":976.35900000000004},"last":17.640999999999998},{"interface":"front","average":{"1min":12.997,"5min":29.919,"15min":23.949000000000002},"min":{"1min":9.1140000000000008,"5min":6.9610000000000003,"15min":5.758},"max":{"1min":18.027999999999999,"5min":509.71699999999998,"15min":509.71699999999998},"last":10.263}]},{"osd":6,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":33.555999999999997,"15min":28.885000000000002},"min":{"1min":10.654999999999999,"5min":7.8869999999999996,"15min":6.3390000000000004},"max":{"1min":22.977,"5min":526.51400000000001,"15min":526.51400000000001},"last":14.087},{"interface":"front","average":{"1min":16.966000000000001,"5min":39.537999999999997,"15min":35.607999999999997},"min":{"1min":12.301,"5min":10.784000000000001,"15min":9.4350000000000005},"max":{"1min":23.710999999999999,"5min":533.20799999999997,"15min":533.20799999999997},"last":19.152000000000001}]},{"osd":7,"last update":"Sat Jan 27 04:19:26 2024","interfaces":[{"interface":"back","average":{"1min":25.797999999999998,"5min":49.715000000000003,"15min":36.329000000000001},"min":{"1min":14.427,"5min":11.739000000000001,"15min":9.5139999999999993},"max":{"1min":74.819000000000003,"5min":977.13800000000003,"15min":977.13800000000003},"last":20.050999999999998},{"interface":"front","average":{"1min":18.117000000000001,"5min":33.683,"15min":29.026},"min":{"1min":11.587999999999999,"5min":8.5850000000000009,"15min":8.5850000000000009},"max":{"1min":52.927999999999997,"5min":549.30100000000004,"15min":549.30100000000004},"last":16.943999999999999}]}]},{"osd":6,"up_from":20,"seq":85899346368,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72180,"kb_used_data":4896,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299660,"statfs":{"total":96636764160,"available":96562851840,"internally_reserved":0,"allocated":5013504,"data_stored":2295222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":9.9730000000000008,"5min":22.446000000000002,"15min":18.524999999999999},"min":{"1min":5.492,"5min":4.6929999999999996,"15min":4.2350000000000003},"max":{"1min":20.303000000000001,"5min":339.68000000000001,"15min":339.68000000000001},"last":9.9979999999999993},{"interface":"front","average":{"1min":14.538,"5min":32.518000000000001,"15min":26.594999999999999},"min":{"1min":9.9329999999999998,"5min":5.5049999999999999,"15min":4.9749999999999996},"max":{"1min":27.638999999999999,"5min":341.03800000000001,"15min":341.03800000000001},"last":21.902999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":12.906000000000001,"5min":24.239999999999998,"15min":18.077999999999999},"min":{"1min":6.1100000000000003,"5min":5.1970000000000001,"15min":4.6950000000000003},"max":{"1min":35.095999999999997,"5min":250.96000000000001,"15min":250.96000000000001},"last":6.9530000000000003},{"interface":"front","average":{"1min":12.737,"5min":29.638999999999999,"15min":23.449000000000002},"min":{"1min":6.181,"5min":5.0679999999999996,"15min":4.7670000000000003},"max":{"1min":27.792000000000002,"5min":350.74599999999998,"15min":350.74599999999998},"last":8.3989999999999991}]},{"osd":2,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":12.606999999999999,"5min":31.344999999999999,"15min":23.207000000000001},"min":{"1min":6.6669999999999998,"5min":6.6669999999999998,"15min":6.0839999999999996},"max":{"1min":25.248999999999999,"5min":349.19299999999998,"15min":349.19299999999998},"last":18.024000000000001},{"interface":"front","average":{"1min":14.715,"5min":35.548000000000002,"15min":29.663},"min":{"1min":9.3729999999999993,"5min":6.5499999999999998,"15min":6.4630000000000001},"max":{"1min":23.071999999999999,"5min":354.80799999999999,"15min":354.80799999999999},"last":24.498999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:43 2024","interfaces":[{"interface":"back","average":{"1min":16.457000000000001,"5min":32.703000000000003,"15min":24.315000000000001},"min":{"1min":6.9299999999999997,"5min":6.9279999999999999,"15min":6.7469999999999999},"max":{"1min":54.694000000000003,"5min":247.446,"15min":265.50299999999999},"last":9.25},{"interface":"front","average":{"1min":20.056999999999999,"5min":34.040999999999997,"15min":28.632999999999999},"min":{"1min":10.635999999999999,"5min":6.4930000000000003,"15min":6.4930000000000003},"max":{"1min":49.180999999999997,"5min":341.68400000000003,"15min":352.79399999999998},"last":26.282}]},{"osd":4,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":15.965,"5min":27.364999999999998,"15min":24.411000000000001},"min":{"1min":10.279999999999999,"5min":6.7590000000000003,"15min":6.5519999999999996},"max":{"1min":33.625999999999998,"5min":243.96100000000001,"15min":266.12400000000002},"last":20.373000000000001},{"interface":"front","average":{"1min":14.163,"5min":30.145,"15min":27.082000000000001},"min":{"1min":7.359,"5min":6.6859999999999999,"15min":6.0289999999999999},"max":{"1min":29.823,"5min":348.11500000000001,"15min":348.11500000000001},"last":18.652999999999999}]},{"osd":5,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":17.280999999999999,"5min":30.981999999999999,"15min":26.800000000000001},"min":{"1min":11.436999999999999,"5min":7.827,"15min":7.827},"max":{"1min":28.574000000000002,"5min":295.56,"15min":295.56},"last":28.443999999999999},{"interface":"front","average":{"1min":19.364000000000001,"5min":37.106000000000002,"15min":30.992000000000001},"min":{"1min":11.782,"5min":10.959,"15min":8.3569999999999993},"max":{"1min":47.966999999999999,"5min":349.81,"15min":354.55099999999999},"last":29.861999999999998}]},{"osd":7,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":26.315000000000001,"5min":38.106000000000002,"15min":25.777000000000001},"min":{"1min":14.762,"5min":9.7609999999999992,"15min":7.4199999999999999},"max":{"1min":93.060000000000002,"5min":346.697,"15min":346.697},"last":26.91},{"interface":"front","average":{"1min":18.989999999999998,"5min":37.228000000000002,"15min":28.795000000000002},"min":{"1min":11.340999999999999,"5min":7.7670000000000003,"15min":7.7670000000000003},"max":{"1min":49.277000000000001,"5min":352.11900000000003,"15min":369.92899999999997},"last":19.280000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346365,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72164,"kb_used_data":4888,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299676,"statfs":{"total":96636764160,"available":96562868224,"internally_reserved":0,"allocated":5005312,"data_stored":2288496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":12.260999999999999,"5min":21.777000000000001,"15min":17.795999999999999},"min":{"1min":4.5209999999999999,"5min":4.4020000000000001,"15min":3.782},"max":{"1min":30.248000000000001,"5min":271.60599999999999,"15min":271.60599999999999},"last":30.248000000000001},{"interface":"front","average":{"1min":11.218999999999999,"5min":19.111999999999998,"15min":15.164999999999999},"min":{"1min":4.8700000000000001,"5min":4.8700000000000001,"15min":4.7830000000000004},"max":{"1min":36.665999999999997,"5min":141.624,"15min":141.624},"last":22.373999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":15.82,"5min":20.286999999999999,"15min":20.946000000000002},"min":{"1min":7.3730000000000002,"5min":5.8159999999999998,"15min":5.8159999999999998},"max":{"1min":44.912999999999997,"5min":228.27600000000001,"15min":228.27600000000001},"last":32.338000000000001},{"interface":"front","average":{"1min":13.35,"5min":23.087,"15min":21.771999999999998},"min":{"1min":7.7060000000000004,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":41.997,"5min":276.93299999999999,"15min":276.93299999999999},"last":28.045000000000002}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":18.422999999999998,"5min":21.227,"15min":22.727},"min":{"1min":7.9669999999999996,"5min":7.8319999999999999,"15min":6.4219999999999997},"max":{"1min":62.054000000000002,"5min":138.13300000000001,"15min":252.38399999999999},"last":30.907},{"interface":"front","average":{"1min":16.931999999999999,"5min":26.645,"15min":22.036999999999999},"min":{"1min":9.7780000000000005,"5min":7.4930000000000003,"15min":5.8979999999999997},"max":{"1min":48.671999999999997,"5min":305.358,"15min":321.09399999999999},"last":23.082999999999998}]},{"osd":3,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":19.376000000000001,"5min":22.815999999999999,"15min":22.629999999999999},"min":{"1min":10.507999999999999,"5min":8.452,"15min":8.1780000000000008},"max":{"1min":61.652999999999999,"5min":89.129999999999995,"15min":195.47800000000001},"last":21.466999999999999},{"interface":"front","average":{"1min":14.627000000000001,"5min":17.146999999999998,"15min":16.039999999999999},"min":{"1min":6.4699999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":91.978999999999999,"5min":120.044,"15min":154.279},"last":7.0910000000000002}]},{"osd":5,"last update":"Sat Jan 27 04:19:38 2024","interfaces":[{"interface":"back","average":{"1min":16.870999999999999,"5min":26.902000000000001,"15min":21.398},"min":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.1779999999999999},"max":{"1min":57.954999999999998,"5min":243.983,"15min":243.983},"last":43.405999999999999},{"interface":"front","average":{"1min":21.181000000000001,"5min":27.506,"15min":24.664000000000001},"min":{"1min":11.605,"5min":9.7690000000000001,"15min":9.7200000000000006},"max":{"1min":61.289000000000001,"5min":188.453,"15min":191.036},"last":59.655000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":21.027000000000001,"5min":29.692,"15min":27.587},"min":{"1min":8.3409999999999993,"5min":8.3409999999999993,"15min":8.3409999999999993},"max":{"1min":46.170999999999999,"5min":259.351,"15min":259.351},"last":8.3409999999999993},{"interface":"front","average":{"1min":24.855,"5min":30.391999999999999,"15min":24.253},"min":{"1min":14.805,"5min":6.9089999999999998,"15min":6.4630000000000001},"max":{"1min":49.533999999999999,"5min":295.00900000000001,"15min":295.00900000000001},"last":25.052}]},{"osd":7,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":20.744,"5min":25.904,"15min":21.736999999999998},"min":{"1min":10.656000000000001,"5min":8.1850000000000005,"15min":6.4269999999999996},"max":{"1min":52.637999999999998,"5min":206.73400000000001,"15min":206.73400000000001},"last":9.5030000000000001},{"interface":"front","average":{"1min":26.753,"5min":32.119,"15min":27.643000000000001},"min":{"1min":15.661,"5min":10.401999999999999,"15min":9.7940000000000005},"max":{"1min":90.658000000000001,"5min":262.28300000000002,"15min":262.28300000000002},"last":29.603999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477181,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72164,"kb_used_data":4884,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299676,"statfs":{"total":96636764160,"available":96562868224,"internally_reserved":0,"allocated":5001216,"data_stored":2288491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:36 2024","interfaces":[{"interface":"back","average":{"1min":11.134,"5min":16.122,"15min":16.141999999999999},"min":{"1min":5.3609999999999998,"5min":5.3609999999999998,"15min":4.6310000000000002},"max":{"1min":33.012,"5min":93.953000000000003,"15min":238.642},"last":13.462},{"interface":"front","average":{"1min":14.760999999999999,"5min":20.975000000000001,"15min":20.623999999999999},"min":{"1min":8.0269999999999992,"5min":7.274,"15min":7.274},"max":{"1min":34.912999999999997,"5min":135.18299999999999,"15min":230.04900000000001},"last":14.769}]},{"osd":1,"last update":"Sat Jan 27 04:19:16 2024","interfaces":[{"interface":"back","average":{"1min":10.414,"5min":15.129,"15min":15.5},"min":{"1min":6.0369999999999999,"5min":5.0730000000000004,"15min":4.7359999999999998},"max":{"1min":27.911000000000001,"5min":103.377,"15min":188.899},"last":6.1779999999999999},{"interface":"front","average":{"1min":10.236000000000001,"5min":16.545000000000002,"15min":20.106000000000002},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":30.829000000000001,"5min":102.009,"15min":218.97399999999999},"last":18.475999999999999}]},{"osd":2,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":16.824000000000002,"5min":22.969999999999999,"15min":21.879999999999999},"min":{"1min":7.8319999999999999,"5min":7.8319999999999999,"15min":7.8319999999999999},"max":{"1min":44.707000000000001,"5min":130.16499999999999,"15min":322.24599999999998},"last":16.512},{"interface":"front","average":{"1min":19.797999999999998,"5min":25.126000000000001,"15min":25.841999999999999},"min":{"1min":11,"5min":9.8699999999999992,"15min":9.8699999999999992},"max":{"1min":53.359000000000002,"5min":201.851,"15min":326.04700000000003},"last":20.082999999999998}]},{"osd":4,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":22.923999999999999,"5min":27.471,"15min":25.827000000000002},"min":{"1min":11.742000000000001,"5min":10.131,"15min":10.131},"max":{"1min":115.759,"5min":220.51300000000001,"15min":250.255},"last":19.363},{"interface":"front","average":{"1min":19.719999999999999,"5min":20.579999999999998,"15min":17.524999999999999},"min":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.5049999999999999},"max":{"1min":150.36199999999999,"5min":221.44999999999999,"15min":227.67400000000001},"last":7.9390000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:19:29 2024","interfaces":[{"interface":"back","average":{"1min":15.736000000000001,"5min":17.591000000000001,"15min":15.472},"min":{"1min":7.2169999999999996,"5min":5.5979999999999999,"15min":4.9359999999999999},"max":{"1min":49.698999999999998,"5min":212.291,"15min":212.291},"last":17.169},{"interface":"front","average":{"1min":16.992999999999999,"5min":19.856999999999999,"15min":19.529},"min":{"1min":8.782,"5min":6.742,"15min":6.3550000000000004},"max":{"1min":70.760000000000005,"5min":147.773,"15min":237.16300000000001},"last":15.65}]},{"osd":6,"last update":"Sat Jan 27 04:19:57 2024","interfaces":[{"interface":"back","average":{"1min":15.177,"5min":20.619,"15min":19.541},"min":{"1min":8.1270000000000007,"5min":7.4939999999999998,"15min":6.1760000000000002},"max":{"1min":27.283999999999999,"5min":145.55199999999999,"15min":261.56799999999998},"last":12.894},{"interface":"front","average":{"1min":16.452000000000002,"5min":23.617999999999999,"15min":23.219999999999999},"min":{"1min":11.131,"5min":7.8700000000000001,"15min":7.8700000000000001},"max":{"1min":34.841999999999999,"5min":135.69900000000001,"15min":276.86099999999999},"last":10.686}]},{"osd":7,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":27.474,"5min":29.882999999999999,"15min":26.277999999999999},"min":{"1min":12.359999999999999,"5min":8.0449999999999999,"15min":8.0449999999999999},"max":{"1min":188.46100000000001,"5min":188.46100000000001,"15min":382.57999999999998},"last":22.079000000000001},{"interface":"front","average":{"1min":24.861000000000001,"5min":28.331,"15min":22.138999999999999},"min":{"1min":9.2249999999999996,"5min":9.2249999999999996,"15min":8.9849999999999994},"max":{"1min":222.232,"5min":222.232,"15min":254.12},"last":10.050000000000001}]}]},{"osd":0,"up_from":16,"seq":68719477179,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72160,"kb_used_data":4884,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299680,"statfs":{"total":96636764160,"available":96562872320,"internally_reserved":0,"allocated":5001216,"data_stored":2288491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":11.346,"5min":21.847999999999999,"15min":19.535},"min":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"max":{"1min":28.620000000000001,"5min":197.15600000000001,"15min":286.24799999999999},"last":9.8729999999999993},{"interface":"front","average":{"1min":13.196999999999999,"5min":29.672999999999998,"15min":23.280999999999999},"min":{"1min":6.726,"5min":6.0350000000000001,"15min":4.7960000000000003},"max":{"1min":36.061,"5min":254.524,"15min":267.99299999999999},"last":8.3680000000000003}]},{"osd":2,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":12.795999999999999,"5min":24.466999999999999,"15min":21.399000000000001},"min":{"1min":6.4829999999999997,"5min":5.984,"15min":5.1449999999999996},"max":{"1min":19.736000000000001,"5min":229.99299999999999,"15min":330.28699999999998},"last":15.855},{"interface":"front","average":{"1min":16.109999999999999,"5min":28.257000000000001,"15min":22.437999999999999},"min":{"1min":10.855,"5min":7.6020000000000003,"15min":6.1669999999999998},"max":{"1min":30.111999999999998,"5min":218.97900000000001,"15min":289.72899999999998},"last":8.9809999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:27 2024","interfaces":[{"interface":"back","average":{"1min":12.762,"5min":23.349,"15min":19.062999999999999},"min":{"1min":6.3230000000000004,"5min":6.3230000000000004,"15min":6.1840000000000002},"max":{"1min":28.111999999999998,"5min":156.90100000000001,"15min":243.779},"last":7.4039999999999999},{"interface":"front","average":{"1min":19.774999999999999,"5min":31.922999999999998,"15min":25.346},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.0860000000000003},"max":{"1min":32.655999999999999,"5min":265.04300000000001,"15min":265.04300000000001},"last":14.35}]},{"osd":4,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":22.652000000000001,"5min":31.033000000000001,"15min":24.866},"min":{"1min":8.2270000000000003,"5min":7.4429999999999996,"15min":7.1870000000000003},"max":{"1min":90.774000000000001,"5min":238.68600000000001,"15min":238.68600000000001},"last":10.619999999999999},{"interface":"front","average":{"1min":26.318999999999999,"5min":30.364999999999998,"15min":27.841000000000001},"min":{"1min":10.281000000000001,"5min":9.7170000000000005,"15min":9.5559999999999992},"max":{"1min":89.200000000000003,"5min":174.03800000000001,"15min":242.22800000000001},"last":16.584}]},{"osd":5,"last update":"Sat Jan 27 04:19:27 2024","interfaces":[{"interface":"back","average":{"1min":17.667999999999999,"5min":40.302999999999997,"15min":26.126999999999999},"min":{"1min":10.754,"5min":7.0670000000000002,"15min":6.665},"max":{"1min":34.997,"5min":385.50799999999998,"15min":385.50799999999998},"last":19.373999999999999},{"interface":"front","average":{"1min":21.75,"5min":42.832999999999998,"15min":28.548999999999999},"min":{"1min":11.871,"5min":9.6639999999999997,"15min":6.5839999999999996},"max":{"1min":47.709000000000003,"5min":337.73700000000002,"15min":337.73700000000002},"last":17.199000000000002}]},{"osd":6,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.641999999999999,"5min":39.985999999999997,"15min":31.425000000000001},"min":{"1min":13.096,"5min":10.368,"15min":10.368},"max":{"1min":36.189999999999998,"5min":236.387,"15min":248.78899999999999},"last":11.795},{"interface":"front","average":{"1min":17.960999999999999,"5min":34.942,"15min":28.195},"min":{"1min":7.9359999999999999,"5min":6.4889999999999999,"15min":6.4889999999999999},"max":{"1min":88.625,"5min":252.99700000000001,"15min":381.90800000000002},"last":14.997999999999999}]},{"osd":7,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.481000000000002,"5min":32.908000000000001,"15min":28.715},"min":{"1min":8.6880000000000006,"5min":8.6880000000000006,"15min":8.4670000000000005},"max":{"1min":56.813000000000002,"5min":232.21600000000001,"15min":236.81299999999999},"last":12.441000000000001},{"interface":"front","average":{"1min":21.489999999999998,"5min":37.271000000000001,"15min":28.934000000000001},"min":{"1min":12.420999999999999,"5min":10.638,"15min":7.2409999999999997},"max":{"1min":34.862000000000002,"5min":249.29499999999999,"15min":249.29499999999999},"last":13.699999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509882,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72540,"kb_used_data":5264,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299300,"statfs":{"total":96636764160,"available":96562483200,"internally_reserved":0,"allocated":5390336,"data_stored":2682863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":21.77,"5min":23.620999999999999,"15min":18.902000000000001},"min":{"1min":5.0140000000000002,"5min":4.4569999999999999,"15min":4.3319999999999999},"max":{"1min":203.28899999999999,"5min":232.74199999999999,"15min":278.56900000000002},"last":5.2510000000000003},{"interface":"front","average":{"1min":19.181999999999999,"5min":22.715,"15min":17.931000000000001},"min":{"1min":7.4649999999999999,"5min":5.4530000000000003,"15min":4.8129999999999997},"max":{"1min":115.354,"5min":234.15799999999999,"15min":305.03800000000001},"last":9.2330000000000005}]},{"osd":1,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":31.167000000000002,"5min":26.742000000000001,"15min":23.835999999999999},"min":{"1min":7.3579999999999997,"5min":6.0549999999999997,"15min":6.0549999999999997},"max":{"1min":192.631,"5min":227.602,"15min":304.28300000000002},"last":14.456},{"interface":"front","average":{"1min":28.222000000000001,"5min":23.238,"15min":20.527000000000001},"min":{"1min":6.5300000000000002,"5min":6.5300000000000002,"15min":5.0199999999999996},"max":{"1min":175.59700000000001,"5min":175.59700000000001,"15min":224.27199999999999},"last":8.3580000000000005}]},{"osd":2,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":23.504999999999999,"5min":30.835000000000001,"15min":29.007000000000001},"min":{"1min":6.375,"5min":6.375,"15min":6.375},"max":{"1min":105.164,"5min":239.91900000000001,"15min":311.17200000000003},"last":17.838000000000001},{"interface":"front","average":{"1min":22.550000000000001,"5min":26.795000000000002,"15min":25.391999999999999},"min":{"1min":7.056,"5min":6.8170000000000002,"15min":6.8170000000000002},"max":{"1min":104.66500000000001,"5min":209.00899999999999,"15min":314.78699999999998},"last":11.32}]},{"osd":3,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":26.381,"5min":25.306000000000001,"15min":20.504000000000001},"min":{"1min":7.7729999999999997,"5min":6.7690000000000001,"15min":6.274},"max":{"1min":179.31299999999999,"5min":187.23099999999999,"15min":274.71899999999999},"last":13.635},{"interface":"front","average":{"1min":24.838000000000001,"5min":28.155999999999999,"15min":24.609999999999999},"min":{"1min":11.1,"5min":8.0670000000000002,"15min":7.3819999999999997},"max":{"1min":108.70099999999999,"5min":175.44800000000001,"15min":233.72800000000001},"last":12.835000000000001}]},{"osd":4,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":24.469000000000001,"5min":26.721,"15min":22.181000000000001},"min":{"1min":8.5410000000000004,"5min":6.4269999999999996,"15min":5.9059999999999997},"max":{"1min":132.90299999999999,"5min":185.755,"15min":219.21899999999999},"last":9.9589999999999996},{"interface":"front","average":{"1min":26.553000000000001,"5min":30.170999999999999,"15min":25.722000000000001},"min":{"1min":9.5739999999999998,"5min":8.7789999999999999,"15min":7.6689999999999996},"max":{"1min":116.852,"5min":174.81,"15min":174.81},"last":15.851000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":25.891999999999999,"5min":31.309000000000001,"15min":27.731999999999999},"min":{"1min":10.345000000000001,"5min":6.9299999999999997,"15min":6.9299999999999997},"max":{"1min":96.703999999999994,"5min":234.95099999999999,"15min":259.803},"last":18.640999999999998},{"interface":"front","average":{"1min":30.484000000000002,"5min":33.567999999999998,"15min":29.574999999999999},"min":{"1min":10.73,"5min":10.73,"15min":8.2859999999999996},"max":{"1min":98.491,"5min":206.55799999999999,"15min":206.55799999999999},"last":17.457999999999998}]},{"osd":7,"last update":"Sat Jan 27 04:20:14 2024","interfaces":[{"interface":"back","average":{"1min":33.548999999999999,"5min":31.448,"15min":26.803000000000001},"min":{"1min":11.276999999999999,"5min":8.0739999999999998,"15min":8.0739999999999998},"max":{"1min":180.07300000000001,"5min":190.39099999999999,"15min":319.75400000000002},"last":20.158999999999999},{"interface":"front","average":{"1min":32.110999999999997,"5min":34.119999999999997,"15min":28.417999999999999},"min":{"1min":10.561999999999999,"5min":9.0800000000000001,"15min":6.6459999999999999},"max":{"1min":217.697,"5min":240.69800000000001,"15min":277.59500000000003},"last":15.109}]}]},{"osd":7,"up_from":13,"seq":55834575291,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72164,"kb_used_data":4888,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299676,"statfs":{"total":96636764160,"available":96562868224,"internally_reserved":0,"allocated":5005312,"data_stored":2288496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:00 2024","interfaces":[{"interface":"back","average":{"1min":10.483000000000001,"5min":21.335000000000001,"15min":15.542999999999999},"min":{"1min":5.649,"5min":4.7370000000000001,"15min":4.1779999999999999},"max":{"1min":24.736999999999998,"5min":162.80799999999999,"15min":162.80799999999999},"last":6.8440000000000003},{"interface":"front","average":{"1min":13.324,"5min":28.123999999999999,"15min":19.064},"min":{"1min":6.5460000000000003,"5min":4.4020000000000001,"15min":4.4020000000000001},"max":{"1min":22.442,"5min":265.62200000000001,"15min":265.62200000000001},"last":11.18}]},{"osd":1,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":14.460000000000001,"5min":25.378,"15min":21.184000000000001},"min":{"1min":9.5470000000000006,"5min":6.0899999999999999,"15min":6.0899999999999999},"max":{"1min":22.472999999999999,"5min":171.16800000000001,"15min":204.17099999999999},"last":11.798999999999999},{"interface":"front","average":{"1min":8.282,"5min":20.821000000000002,"15min":18.806999999999999},"min":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"max":{"1min":20.916,"5min":192.11000000000001,"15min":323.48599999999999},"last":14.613}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.647,"5min":30.683,"15min":21.582999999999998},"min":{"1min":9.0679999999999996,"5min":6.1779999999999999,"15min":6.1269999999999998},"max":{"1min":23.294,"5min":420.65199999999999,"15min":420.65199999999999},"last":9.3480000000000008},{"interface":"front","average":{"1min":13.297000000000001,"5min":28.969000000000001,"15min":22.788},"min":{"1min":6.25,"5min":6.25,"15min":6.25},"max":{"1min":36.289000000000001,"5min":453.39100000000002,"15min":453.39100000000002},"last":13.087999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":14.026,"5min":27.434000000000001,"15min":23.035},"min":{"1min":9.8599999999999994,"5min":8.0830000000000002,"15min":7.6890000000000001},"max":{"1min":23.341999999999999,"5min":226.72300000000001,"15min":226.72300000000001},"last":8.4209999999999994},{"interface":"front","average":{"1min":15.734999999999999,"5min":24.466000000000001,"15min":21.619},"min":{"1min":7.6680000000000001,"5min":5.7270000000000003,"15min":5.7270000000000003},"max":{"1min":23.318999999999999,"5min":197.01499999999999,"15min":197.01499999999999},"last":12.484}]},{"osd":4,"last update":"Sat Jan 27 04:20:11 2024","interfaces":[{"interface":"back","average":{"1min":15.734,"5min":29.295000000000002,"15min":22.515000000000001},"min":{"1min":9.2780000000000005,"5min":8.4749999999999996,"15min":8.3200000000000003},"max":{"1min":48.780000000000001,"5min":227.72300000000001,"15min":227.72300000000001},"last":10.061999999999999},{"interface":"front","average":{"1min":18.984000000000002,"5min":32.296999999999997,"15min":24.923999999999999},"min":{"1min":11.102,"5min":9.3059999999999992,"15min":9.3059999999999992},"max":{"1min":55.207000000000001,"5min":230.16,"15min":230.16},"last":18.850000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:19:22 2024","interfaces":[{"interface":"back","average":{"1min":18.823,"5min":38.701000000000001,"15min":28.065000000000001},"min":{"1min":14.417,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":49.390999999999998,"5min":276.23500000000001,"15min":276.23500000000001},"last":13.885},{"interface":"front","average":{"1min":21.015999999999998,"5min":35.655000000000001,"15min":29.832999999999998},"min":{"1min":16.212,"5min":10.522,"15min":10.522},"max":{"1min":36.661000000000001,"5min":408.774,"15min":408.774},"last":18.050000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":20.045999999999999,"5min":34.045000000000002,"15min":29.82},"min":{"1min":15.138,"5min":10.936,"15min":10.936},"max":{"1min":27.131,"5min":276.92099999999999,"15min":304.952},"last":17.254000000000001},{"interface":"front","average":{"1min":14.42,"5min":30.312000000000001,"15min":26.094000000000001},"min":{"1min":7.2199999999999998,"5min":7.2199999999999998,"15min":7.2199999999999998},"max":{"1min":25.401,"5min":302.61700000000002,"15min":302.61700000000002},"last":15.374000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T04:20:17.890 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T04:20:17.890 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:03.843 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:17.909 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:03.865 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:17.933 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:03.878 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:17.934 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:03.880 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:17.938 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:03.894 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:17.943 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:03.899 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:17.948 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:03.894 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:17.958 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:03.914 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.994+0000 7fe4bba0e640 10 monclient: get_monmap_and_config 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.994+0000 7fe4bba0e640 10 monclient: build_initial_monmap 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.994+0000 7fe4bba0e640 10 monclient: monmap: 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:17.995190+0000 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:17.995190+0000 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:17.995 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:17.996 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:17.996 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:17.996 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:17.996 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:17.996 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:17.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.994+0000 7fe4bba0e640 1 Processor -- start 2024-01-27T04:20:17.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.994+0000 7fe4bba0e640 1 -- start start 2024-01-27T04:20:17.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.994+0000 7fe4bba0e640 10 monclient: init 2024-01-27T04:20:17.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:17.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-01-27T04:20:17.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4b40b4680 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:17.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40b4a60 0x7fe4b40b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:17.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b40b4a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:17.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe4b40b3b30 0x7fe4b40b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:17.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b40b3b30 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:17.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b40afbe0 con 0x7fe4b40b4680 2024-01-27T04:20:17.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:17.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b4142240 con 0x7fe4b40b3b30 2024-01-27T04:20:17.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:17.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40b0890 con 0x7fe4b40b4a60 2024-01-27T04:20:17.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:17.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:17.995+0000 7fe4bba0e640 10 monclient(hunting): authenticate will time out at 3701.349367s 2024-01-27T04:20:18.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.036+0000 7fe4b8fa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40b4a60 0x7fe4b40b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.039+0000 7fe4b8fa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40b4a60 0x7fe4b40b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:41078/0 (socket says 172.21.15.110:41078) 2024-01-27T04:20:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.039+0000 7fe4b8fa9640 1 -- 172.21.15.110:0/1890161404 learned_addr learned my addr 172.21.15.110:0/1890161404 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.039+0000 7fe4b8fa9640 10 monclient(hunting): get_auth_request con 0x7fe4b40b4a60 auth_method 0 2024-01-27T04:20:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.039+0000 7fe4b8fa9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.039+0000 7fe4b8fa9640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:18.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.039+0000 7fe4b8fa9640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:18.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.067+0000 7fe4b9fab640 1 --2- 172.21.15.110:0/1890161404 >> v2:172.21.15.162:3300/0 conn(0x7fe4b40b3b30 0x7fe4b40b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.071+0000 7fe4b9fab640 10 monclient(hunting): get_auth_request con 0x7fe4b40b3b30 auth_method 0 2024-01-27T04:20:18.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.071+0000 7fe4b9fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:18.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.071+0000 7fe4b9fab640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:18.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.071+0000 7fe4b9fab640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:18.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.073+0000 7fe4b9fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:18.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.073+0000 7fe4b9fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:18.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.073+0000 7fe4b9fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:18.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.074+0000 7fe4b8fa9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:18.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.074+0000 7fe4b8fa9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:18.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.074+0000 7fe4b8fa9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:18.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.077+0000 7fe4b8fa9640 10 monclient(hunting): handle_auth_done global_id 18851 payload 274 2024-01-27T04:20:18.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.077+0000 7fe4b8fa9640 1 -- 172.21.15.110:0/1890161404 >> v2:172.21.15.162:3300/0 conn(0x7fe4b40b3b30 msgr2=0x7fe4b40b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:18.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.077+0000 7fe4b8fa9640 1 --2- 172.21.15.110:0/1890161404 >> v2:172.21.15.162:3300/0 conn(0x7fe4b40b3b30 0x7fe4b40b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.077+0000 7fe4b8fa9640 1 -- 172.21.15.110:0/1890161404 >> v1:172.21.15.162:6789/0 conn(0x7fe4b40b4680 legacy=0x7fe4b4142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:18.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.077+0000 7fe4b8fa9640 10 monclient: _finish_hunting 0 2024-01-27T04:20:18.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.077+0000 7fe4b8fa9640 1 monclient: found mon.noname-b 2024-01-27T04:20:18.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.077+0000 7fe4b8fa9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:18.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.077+0000 7fe4b8fa9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:18.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.077+0000 7fe4b8fa9640 1 -- 172.21.15.110:0/1890161404 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b40b0b00 con 0x7fe4b40b4a60 2024-01-27T04:20:18.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.105+0000 7fe4b8fa9640 1 --2- 172.21.15.110:0/1890161404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40b4a60 0x7fe4b40b1470 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009870 tx=0x7fe4a402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5706ce45a3be363 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:18.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 1 -- 172.21.15.110:0/1890161404 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a40317c0 con 0x7fe4b40b4a60 2024-01-27T04:20:18.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:18.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:18.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 10 monclient: dump: 2024-01-27T04:20:18.139 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:18.139 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:18.139 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.139 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.139 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 10 monclient: _finish_auth 0 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 10 monclient: _check_auth_tickets 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 1 -- 172.21.15.110:0/1890161404 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a4031920 con 0x7fe4b40b4a60 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 1 -- 172.21.15.110:0/1890161404 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4031cb0 con 0x7fe4b40b4a60 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4bba0e640 5 monclient: authenticate success, global_id 18851 2024-01-27T04:20:18.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4bba0e640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4bba0e640 4 set_mon_vals no callback set 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.137+0000 7fe4abfff640 10 monclient: dump: 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:18.141 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:18.142 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 10 monclient: shutdown 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 1 -- 172.21.15.110:0/1890161404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40b4a60 msgr2=0x7fe4b40b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 1 --2- 172.21.15.110:0/1890161404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40b4a60 0x7fe4b40b1470 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009870 tx=0x7fe4a402b860 comp rx=0 tx=0).stop 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 1 -- 172.21.15.110:0/1890161404 shutdown_connections 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 1 --2- 172.21.15.110:0/1890161404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40b4a60 0x7fe4b40b1470 unknown :-1 s=CLOSED pgs=3405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 1 --2- 172.21.15.110:0/1890161404 >> v2:172.21.15.162:3300/0 conn(0x7fe4b40b3b30 0x7fe4b40b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 1 -- 172.21.15.110:0/1890161404 >> 172.21.15.110:0/1890161404 conn(0x7fe4b4066490 msgr2=0x7fe4b40668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 1 -- 172.21.15.110:0/1890161404 shutdown_connections 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.138+0000 7fe4bba0e640 1 -- 172.21.15.110:0/1890161404 wait complete. 2024-01-27T04:20:18.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.139+0000 7fe4bba0e640 10 monclient: build_initial_monmap 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.139+0000 7fe4bba0e640 10 monclient: monmap: 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.139+0000 7fe4bba0e640 1 librados: starting msgr at 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.139+0000 7fe4bba0e640 1 librados: starting objecter 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.139+0000 7fe4bba0e640 1 Processor -- start 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.139+0000 7fe4bba0e640 1 -- start start 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.139+0000 7fe4bba0e640 1 librados: setting wanted keys 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.139+0000 7fe4bba0e640 1 librados: calling monclient init 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.139+0000 7fe4bba0e640 10 monclient: init 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b40b3b30 0x7fe4b40ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b40b3b30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b40b4680 0x7fe4b413ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b40b4680 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40aac10 0x7fe4b40ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b40aac10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b4137c70 con 0x7fe4b40b3b30 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40acf90 con 0x7fe4b40b4680 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40ab0c0 con 0x7fe4b40aac10 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:18.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.140+0000 7fe4bba0e640 10 monclient(hunting): authenticate will time out at 3701.494653s 2024-01-27T04:20:18.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.143+0000 7fe4b9fab640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40aac10 0x7fe4b40ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.150+0000 7fe4b97aa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b40b3b30 0x7fe4b40ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.155+0000 7fe4b97aa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b40b3b30 0x7fe4b40ad690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40748/0 (socket says 172.21.15.110:40748) 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.155+0000 7fe4b97aa640 1 -- 172.21.15.110:0/2440409953 learned_addr learned my addr 172.21.15.110:0/2440409953 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.155+0000 7fe4b97aa640 10 monclient(hunting): get_auth_request con 0x7fe4b40b3b30 auth_method 0 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.155+0000 7fe4b97aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.155+0000 7fe4b97aa640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.155+0000 7fe4b97aa640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.156+0000 7fe4b9fab640 10 monclient(hunting): get_auth_request con 0x7fe4b40aac10 auth_method 0 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.156+0000 7fe4b9fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.156+0000 7fe4b9fab640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:18.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.156+0000 7fe4b9fab640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:18.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.157+0000 7fe4b9fab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:18.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.157+0000 7fe4b9fab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:18.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.157+0000 7fe4b9fab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:18.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.159+0000 7fe4b97aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:18.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.159+0000 7fe4b97aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:18.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.159+0000 7fe4b97aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:18.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 10 monclient(hunting): handle_auth_done global_id 16983 payload 824 2024-01-27T04:20:18.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 1 -- 172.21.15.110:0/2440409953 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40aac10 msgr2=0x7fe4b40ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:18.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40aac10 0x7fe4b40ab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 1 -- 172.21.15.110:0/2440409953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b40b4680 msgr2=0x7fe4b413ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:18.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b40b4680 0x7fe4b413ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 10 monclient: _finish_hunting 0 2024-01-27T04:20:18.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 1 monclient: found mon.noname-a 2024-01-27T04:20:18.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:18.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T04:20:18.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.164+0000 7fe4b97aa640 1 -- 172.21.15.110:0/2440409953 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40094d0 con 0x7fe4b40b3b30 2024-01-27T04:20:18.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.180+0000 7fe4b97aa640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b40b3b30 0x7fe4b40ad690 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000c430 tx=0x7fe4b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:18.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.180+0000 7fe4a9ffb640 10 client.?.objecter ms_handle_connect 0x7fe4b40b3b30 2024-01-27T04:20:18.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.180+0000 7fe4a9ffb640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:18.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4a9ffb640 1 -- 172.21.15.110:0/2440409953 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0011790 con 0x7fe4b40b3b30 2024-01-27T04:20:18.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:18.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T04:20:18.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4a9ffb640 10 monclient: dump: 2024-01-27T04:20:18.218 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:18.218 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4a9ffb640 10 monclient: _finish_auth 0 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4a9ffb640 10 monclient: _check_auth_tickets 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4bba0e640 5 monclient: authenticate success, global_id 16983 2024-01-27T04:20:18.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4bba0e640 10 monclient: _renew_subs 2024-01-27T04:20:18.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4bba0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:18.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b40ad270 con 0x7fe4b40b3b30 2024-01-27T04:20:18.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4bba0e640 10 client.16983.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:18.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4bba0e640 10 monclient: _renew_subs 2024-01-27T04:20:18.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4bba0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:18.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b40abfd0 con 0x7fe4b40b3b30 2024-01-27T04:20:18.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.216+0000 7fe4bba0e640 1 librados: init done 2024-01-27T04:20:18.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.217+0000 7fe4bba0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:18.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.217+0000 7fe4bba0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:18.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.217+0000 7fe4bba0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:18.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.217+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b4074d10 con 0x7fe4b40b3b30 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.224+0000 7fe4a9ffb640 1 -- 172.21.15.110:0/2440409953 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b0003800 con 0x7fe4b40b3b30 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.224+0000 7fe4a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.224+0000 7fe4a9ffb640 1 -- 172.21.15.110:0/2440409953 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0011af0 con 0x7fe4b40b3b30 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.224+0000 7fe4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.224+0000 7fe4a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.224+0000 7fe4a9ffb640 10 monclient: dump: 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.225 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.226 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:18.226 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.226 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:18.226 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:18.226 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:18.226 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.224+0000 7fe4ab7fe640 4 set_mon_vals no callback set 2024-01-27T04:20:18.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.297+0000 7fe4a9ffb640 1 -- 172.21.15.110:0/2440409953 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fe4b0011d10 con 0x7fe4b40b3b30 2024-01-27T04:20:18.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.297+0000 7fe4a9ffb640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe49806c9a0 0x7fe49806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.297+0000 7fe4b8fa9640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe49806c9a0 0x7fe49806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.297+0000 7fe4b8fa9640 10 monclient: get_auth_request con 0x7fe49806c9a0 auth_method 0 2024-01-27T04:20:18.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.298+0000 7fe4a9ffb640 10 client.16983.objecter ms_handle_connect 0x7fe49806c9a0 2024-01-27T04:20:18.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.298+0000 7fe4b8fa9640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe49806c9a0 0x7fe49806ee60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4036000 tx=0x7fe4a4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.549+0000 7fe4a9ffb640 1 -- 172.21.15.110:0/2440409953 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7fe4b0059440 con 0x7fe4b40b3b30 2024-01-27T04:20:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.549+0000 7fe4a9ffb640 10 client.16983.objecter ms_dispatch 0x7fe4b4000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.549+0000 7fe4a9ffb640 3 client.16983.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.549+0000 7fe4a9ffb640 3 client.16983.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.549+0000 7fe4a9ffb640 20 client.16983.objecter dump_active .. 0 homeless 2024-01-27T04:20:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.549+0000 7fe4a9ffb640 1 -- 172.21.15.110:0/2440409953 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fe4b0058b10 con 0x7fe4b40b3b30 2024-01-27T04:20:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.549+0000 7fe4a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:18.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.549+0000 7fe4a9ffb640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:18.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.744+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4b414caf0 con 0x7fe49806c9a0 2024-01-27T04:20:18.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.749+0000 7fe4a9ffb640 1 -- 172.21.15.110:0/2440409953 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151868 (secure 0 0 0) 0x7fe4a4031b10 con 0x7fe49806c9a0 2024-01-27T04:20:18.752 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T04:20:18.753 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 10 librados: watch_flush enter 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 10 librados: watch_flush exit 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 20 client.16983.objecter shutdown clearing up homeless session... 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 10 client.16983.objecter successfully canceled tick 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe49806c9a0 msgr2=0x7fe49806ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe49806c9a0 0x7fe49806ee60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4036000 tx=0x7fe4a4030000 comp rx=0 tx=0).stop 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 10 monclient: shutdown 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b40b3b30 msgr2=0x7fe4b40ad690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b40b3b30 0x7fe4b40ad690 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000c430 tx=0x7fe4b000c8f0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 shutdown_connections 2024-01-27T04:20:18.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe49806c9a0 0x7fe49806ee60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4b40aac10 0x7fe4b40ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b40b4680 0x7fe4b413ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 --2- 172.21.15.110:0/2440409953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4b40b3b30 0x7fe4b40ad690 unknown :-1 s=CLOSED pgs=3460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 >> 172.21.15.110:0/2440409953 conn(0x7fe4b4066490 msgr2=0x7fe4b4066870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:18.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 shutdown_connections 2024-01-27T04:20:18.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 -- 172.21.15.110:0/2440409953 wait complete. 2024-01-27T04:20:18.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.753+0000 7fe4bba0e640 1 librados: shutdown 2024-01-27T04:20:18.767 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":2451,"stamp":"2024-01-27T04:20:17.553087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":578036,"kb_used_data":39808,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754396684,"statfs":{"total":773094113280,"available":772502204416,"internally_reserved":0,"allocated":40763392,"data_stored":19076953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":151,"apply_latency_ms":151,"commit_latency_ns":151000000,"apply_latency_ns":151000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006263"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2557,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.713917+0000","last_change":"2024-01-27T03:45:12.104231+0000","last_active":"2024-01-27T04:20:16.713917+0000","last_peered":"2024-01-27T04:20:16.713917+0000","last_clean":"2024-01-27T04:20:16.713917+0000","last_became_active":"2024-01-27T03:45:12.095646+0000","last_became_peered":"2024-01-27T03:45:12.095646+0000","last_unstale":"2024-01-27T04:20:16.713917+0000","last_undegraded":"2024-01-27T04:20:16.713917+0000","last_fullsized":"2024-01-27T04:20:16.713917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:14:48.806131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2561,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.335104+0000","last_change":"2024-01-27T03:45:50.347781+0000","last_active":"2024-01-27T04:20:12.335104+0000","last_peered":"2024-01-27T04:20:12.335104+0000","last_clean":"2024-01-27T04:20:12.335104+0000","last_became_active":"2024-01-27T03:45:50.339311+0000","last_became_peered":"2024-01-27T03:45:50.339311+0000","last_unstale":"2024-01-27T04:20:12.335104+0000","last_undegraded":"2024-01-27T04:20:12.335104+0000","last_fullsized":"2024-01-27T04:20:12.335104+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:32.762753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2560,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.761038+0000","last_change":"2024-01-27T03:45:01.242765+0000","last_active":"2024-01-27T04:20:15.761038+0000","last_peered":"2024-01-27T04:20:15.761038+0000","last_clean":"2024-01-27T04:20:15.761038+0000","last_became_active":"2024-01-27T03:45:01.173317+0000","last_became_peered":"2024-01-27T03:45:01.173317+0000","last_unstale":"2024-01-27T04:20:15.761038+0000","last_undegraded":"2024-01-27T04:20:15.761038+0000","last_fullsized":"2024-01-27T04:20:15.761038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:45:07.062762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2547,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.781859+0000","last_change":"2024-01-27T03:45:08.326818+0000","last_active":"2024-01-27T04:20:13.781859+0000","last_peered":"2024-01-27T04:20:13.781859+0000","last_clean":"2024-01-27T04:20:13.781859+0000","last_became_active":"2024-01-27T03:45:08.187722+0000","last_became_peered":"2024-01-27T03:45:08.187722+0000","last_unstale":"2024-01-27T04:20:13.781859+0000","last_undegraded":"2024-01-27T04:20:13.781859+0000","last_fullsized":"2024-01-27T04:20:13.781859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:07:16.791137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2548,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.875693+0000","last_change":"2024-01-27T03:45:03.762871+0000","last_active":"2024-01-27T04:20:13.875693+0000","last_peered":"2024-01-27T04:20:13.875693+0000","last_clean":"2024-01-27T04:20:13.875693+0000","last_became_active":"2024-01-27T03:45:03.757221+0000","last_became_peered":"2024-01-27T03:45:03.757221+0000","last_unstale":"2024-01-27T04:20:13.875693+0000","last_undegraded":"2024-01-27T04:20:13.875693+0000","last_fullsized":"2024-01-27T04:20:13.875693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:52.156074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2572,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.736771+0000","last_change":"2024-01-27T03:45:01.929664+0000","last_active":"2024-01-27T04:20:14.736771+0000","last_peered":"2024-01-27T04:20:14.736771+0000","last_clean":"2024-01-27T04:20:14.736771+0000","last_became_active":"2024-01-27T03:45:01.916892+0000","last_became_peered":"2024-01-27T03:45:01.916892+0000","last_unstale":"2024-01-27T04:20:14.736771+0000","last_undegraded":"2024-01-27T04:20:14.736771+0000","last_fullsized":"2024-01-27T04:20:14.736771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:02:49.278494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2561,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.352601+0000","last_change":"2024-01-27T03:45:50.416152+0000","last_active":"2024-01-27T04:20:12.352601+0000","last_peered":"2024-01-27T04:20:12.352601+0000","last_clean":"2024-01-27T04:20:12.352601+0000","last_became_active":"2024-01-27T03:45:50.401687+0000","last_became_peered":"2024-01-27T03:45:50.401687+0000","last_unstale":"2024-01-27T04:20:12.352601+0000","last_undegraded":"2024-01-27T04:20:12.352601+0000","last_fullsized":"2024-01-27T04:20:12.352601+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:34:59.292181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2563,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.357036+0000","last_change":"2024-01-27T03:45:16.267960+0000","last_active":"2024-01-27T04:20:12.357036+0000","last_peered":"2024-01-27T04:20:12.357036+0000","last_clean":"2024-01-27T04:20:12.357036+0000","last_became_active":"2024-01-27T03:45:16.039935+0000","last_became_peered":"2024-01-27T03:45:16.039935+0000","last_unstale":"2024-01-27T04:20:12.357036+0000","last_undegraded":"2024-01-27T04:20:12.357036+0000","last_fullsized":"2024-01-27T04:20:12.357036+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:02:05.330053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"48'7","reported_seq":2588,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.717267+0000","last_change":"2024-01-27T03:45:00.305112+0000","last_active":"2024-01-27T04:20:14.717267+0000","last_peered":"2024-01-27T04:20:14.717267+0000","last_clean":"2024-01-27T04:20:14.717267+0000","last_became_active":"2024-01-27T03:45:00.291987+0000","last_became_peered":"2024-01-27T03:45:00.291987+0000","last_unstale":"2024-01-27T04:20:14.717267+0000","last_undegraded":"2024-01-27T04:20:14.717267+0000","last_fullsized":"2024-01-27T04:20:14.717267+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:37.876016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2550,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.745980+0000","last_change":"2024-01-27T03:44:59.103125+0000","last_active":"2024-01-27T04:20:13.745980+0000","last_peered":"2024-01-27T04:20:13.745980+0000","last_clean":"2024-01-27T04:20:13.745980+0000","last_became_active":"2024-01-27T03:44:59.080371+0000","last_became_peered":"2024-01-27T03:44:59.080371+0000","last_unstale":"2024-01-27T04:20:13.745980+0000","last_undegraded":"2024-01-27T04:20:13.745980+0000","last_fullsized":"2024-01-27T04:20:13.745980+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:17.291036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2538,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.585376+0000","last_change":"2024-01-27T03:45:05.767658+0000","last_active":"2024-01-27T04:20:15.585376+0000","last_peered":"2024-01-27T04:20:15.585376+0000","last_clean":"2024-01-27T04:20:15.585376+0000","last_became_active":"2024-01-27T03:45:05.758355+0000","last_became_peered":"2024-01-27T03:45:05.758355+0000","last_unstale":"2024-01-27T04:20:15.585376+0000","last_undegraded":"2024-01-27T04:20:15.585376+0000","last_fullsized":"2024-01-27T04:20:15.585376+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:29:16.853701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2555,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.721456+0000","last_change":"2024-01-27T03:45:11.894875+0000","last_active":"2024-01-27T04:20:16.721456+0000","last_peered":"2024-01-27T04:20:16.721456+0000","last_clean":"2024-01-27T04:20:16.721456+0000","last_became_active":"2024-01-27T03:45:11.884947+0000","last_became_peered":"2024-01-27T03:45:11.884947+0000","last_unstale":"2024-01-27T04:20:16.721456+0000","last_undegraded":"2024-01-27T04:20:16.721456+0000","last_fullsized":"2024-01-27T04:20:16.721456+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:23:21.445636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":2577,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.279707+0000","last_change":"2024-01-27T03:43:25.035922+0000","last_active":"2024-01-27T04:20:12.279707+0000","last_peered":"2024-01-27T04:20:12.279707+0000","last_clean":"2024-01-27T04:20:12.279707+0000","last_became_active":"2024-01-27T03:43:24.404196+0000","last_became_peered":"2024-01-27T03:43:24.404196+0000","last_unstale":"2024-01-27T04:20:12.279707+0000","last_undegraded":"2024-01-27T04:20:12.279707+0000","last_fullsized":"2024-01-27T04:20:12.279707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_clean_scrub_stamp":"2024-01-27T03:43:13.902947+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:56:03.692684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2581,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.460782+0000","last_change":"2024-01-27T03:43:30.313647+0000","last_active":"2024-01-27T04:20:16.460782+0000","last_peered":"2024-01-27T04:20:16.460782+0000","last_clean":"2024-01-27T04:20:16.460782+0000","last_became_active":"2024-01-27T03:43:30.303196+0000","last_became_peered":"2024-01-27T03:43:30.303196+0000","last_unstale":"2024-01-27T04:20:16.460782+0000","last_undegraded":"2024-01-27T04:20:16.460782+0000","last_fullsized":"2024-01-27T04:20:16.460782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:45:18.950474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2549,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.750175+0000","last_change":"2024-01-27T03:45:06.207200+0000","last_active":"2024-01-27T04:20:13.750175+0000","last_peered":"2024-01-27T04:20:13.750175+0000","last_clean":"2024-01-27T04:20:13.750175+0000","last_became_active":"2024-01-27T03:45:06.197607+0000","last_became_peered":"2024-01-27T03:45:06.197607+0000","last_unstale":"2024-01-27T04:20:13.750175+0000","last_undegraded":"2024-01-27T04:20:13.750175+0000","last_fullsized":"2024-01-27T04:20:13.750175+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:30:59.795065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2583,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.446781+0000","last_change":"2024-01-27T03:43:30.123281+0000","last_active":"2024-01-27T04:20:16.446781+0000","last_peered":"2024-01-27T04:20:16.446781+0000","last_clean":"2024-01-27T04:20:16.446781+0000","last_became_active":"2024-01-27T03:43:30.033663+0000","last_became_peered":"2024-01-27T03:43:30.033663+0000","last_unstale":"2024-01-27T04:20:16.446781+0000","last_undegraded":"2024-01-27T04:20:16.446781+0000","last_fullsized":"2024-01-27T04:20:16.446781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:03.978730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2548,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.773182+0000","last_change":"2024-01-27T03:45:07.001823+0000","last_active":"2024-01-27T04:20:13.773182+0000","last_peered":"2024-01-27T04:20:13.773182+0000","last_clean":"2024-01-27T04:20:13.773182+0000","last_became_active":"2024-01-27T03:45:06.986316+0000","last_became_peered":"2024-01-27T03:45:06.986316+0000","last_unstale":"2024-01-27T04:20:13.773182+0000","last_undegraded":"2024-01-27T04:20:13.773182+0000","last_fullsized":"2024-01-27T04:20:13.773182+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:42.117379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2584,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.718217+0000","last_change":"2024-01-27T03:43:30.100170+0000","last_active":"2024-01-27T04:20:14.718217+0000","last_peered":"2024-01-27T04:20:14.718217+0000","last_clean":"2024-01-27T04:20:14.718217+0000","last_became_active":"2024-01-27T03:43:30.072010+0000","last_became_peered":"2024-01-27T03:43:30.072010+0000","last_unstale":"2024-01-27T04:20:14.718217+0000","last_undegraded":"2024-01-27T04:20:14.718217+0000","last_fullsized":"2024-01-27T04:20:14.718217+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:17:46.003541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2553,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.294627+0000","last_change":"2024-01-27T03:45:28.248666+0000","last_active":"2024-01-27T04:20:12.294627+0000","last_peered":"2024-01-27T04:20:12.294627+0000","last_clean":"2024-01-27T04:20:12.294627+0000","last_became_active":"2024-01-27T03:45:28.239949+0000","last_became_peered":"2024-01-27T03:45:28.239949+0000","last_unstale":"2024-01-27T04:20:12.294627+0000","last_undegraded":"2024-01-27T04:20:12.294627+0000","last_fullsized":"2024-01-27T04:20:12.294627+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:11:52.211119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.458866+0000","last_change":"2024-01-27T03:43:30.335032+0000","last_active":"2024-01-27T04:20:16.458866+0000","last_peered":"2024-01-27T04:20:16.458866+0000","last_clean":"2024-01-27T04:20:16.458866+0000","last_became_active":"2024-01-27T03:43:30.329967+0000","last_became_peered":"2024-01-27T03:43:30.329967+0000","last_unstale":"2024-01-27T04:20:16.458866+0000","last_undegraded":"2024-01-27T04:20:16.458866+0000","last_fullsized":"2024-01-27T04:20:16.458866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:30.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2549,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.741986+0000","last_change":"2024-01-27T03:45:03.469717+0000","last_active":"2024-01-27T04:20:13.741986+0000","last_peered":"2024-01-27T04:20:13.741986+0000","last_clean":"2024-01-27T04:20:13.741986+0000","last_became_active":"2024-01-27T03:45:03.435938+0000","last_became_peered":"2024-01-27T03:45:03.435938+0000","last_unstale":"2024-01-27T04:20:13.741986+0000","last_undegraded":"2024-01-27T04:20:13.741986+0000","last_fullsized":"2024-01-27T04:20:13.741986+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:31:47.895364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2559,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.872098+0000","last_change":"2024-01-27T03:43:30.340472+0000","last_active":"2024-01-27T04:20:13.872098+0000","last_peered":"2024-01-27T04:20:13.872098+0000","last_clean":"2024-01-27T04:20:13.872098+0000","last_became_active":"2024-01-27T03:43:30.329560+0000","last_became_peered":"2024-01-27T03:43:30.329560+0000","last_unstale":"2024-01-27T04:20:13.872098+0000","last_undegraded":"2024-01-27T04:20:13.872098+0000","last_fullsized":"2024-01-27T04:20:13.872098+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:10:58.217001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2560,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.350888+0000","last_change":"2024-01-27T03:45:11.682176+0000","last_active":"2024-01-27T04:20:12.350888+0000","last_peered":"2024-01-27T04:20:12.350888+0000","last_clean":"2024-01-27T04:20:12.350888+0000","last_became_active":"2024-01-27T03:45:11.344546+0000","last_became_peered":"2024-01-27T03:45:11.344546+0000","last_unstale":"2024-01-27T04:20:12.350888+0000","last_undegraded":"2024-01-27T04:20:12.350888+0000","last_fullsized":"2024-01-27T04:20:12.350888+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:35:01.350866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2581,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.473814+0000","last_change":"2024-01-27T03:43:30.077908+0000","last_active":"2024-01-27T04:20:16.473814+0000","last_peered":"2024-01-27T04:20:16.473814+0000","last_clean":"2024-01-27T04:20:16.473814+0000","last_became_active":"2024-01-27T03:43:30.057792+0000","last_became_peered":"2024-01-27T03:43:30.057792+0000","last_unstale":"2024-01-27T04:20:16.473814+0000","last_undegraded":"2024-01-27T04:20:16.473814+0000","last_fullsized":"2024-01-27T04:20:16.473814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:05.446156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2562,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.345492+0000","last_change":"2024-01-27T03:45:12.756405+0000","last_active":"2024-01-27T04:20:12.345492+0000","last_peered":"2024-01-27T04:20:12.345492+0000","last_clean":"2024-01-27T04:20:12.345492+0000","last_became_active":"2024-01-27T03:45:12.751200+0000","last_became_peered":"2024-01-27T03:45:12.751200+0000","last_unstale":"2024-01-27T04:20:12.345492+0000","last_undegraded":"2024-01-27T04:20:12.345492+0000","last_fullsized":"2024-01-27T04:20:12.345492+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:07:39.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2571,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.756811+0000","last_change":"2024-01-27T03:43:30.372081+0000","last_active":"2024-01-27T04:20:15.756811+0000","last_peered":"2024-01-27T04:20:15.756811+0000","last_clean":"2024-01-27T04:20:15.756811+0000","last_became_active":"2024-01-27T03:43:30.335129+0000","last_became_peered":"2024-01-27T03:43:30.335129+0000","last_unstale":"2024-01-27T04:20:15.756811+0000","last_undegraded":"2024-01-27T04:20:15.756811+0000","last_fullsized":"2024-01-27T04:20:15.756811+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:01.568710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2570,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.472197+0000","last_change":"2024-01-27T03:45:11.743478+0000","last_active":"2024-01-27T04:20:16.472197+0000","last_peered":"2024-01-27T04:20:16.472197+0000","last_clean":"2024-01-27T04:20:16.472197+0000","last_became_active":"2024-01-27T03:45:11.734547+0000","last_became_peered":"2024-01-27T03:45:11.734547+0000","last_unstale":"2024-01-27T04:20:16.472197+0000","last_undegraded":"2024-01-27T04:20:16.472197+0000","last_fullsized":"2024-01-27T04:20:16.472197+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:58.163231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2579,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.468607+0000","last_change":"2024-01-27T03:43:30.350497+0000","last_active":"2024-01-27T04:20:16.468607+0000","last_peered":"2024-01-27T04:20:16.468607+0000","last_clean":"2024-01-27T04:20:16.468607+0000","last_became_active":"2024-01-27T03:43:30.345375+0000","last_became_peered":"2024-01-27T03:43:30.345375+0000","last_unstale":"2024-01-27T04:20:16.468607+0000","last_undegraded":"2024-01-27T04:20:16.468607+0000","last_fullsized":"2024-01-27T04:20:16.468607+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:58.675410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2570,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.735085+0000","last_change":"2024-01-27T03:45:08.095965+0000","last_active":"2024-01-27T04:20:14.735085+0000","last_peered":"2024-01-27T04:20:14.735085+0000","last_clean":"2024-01-27T04:20:14.735085+0000","last_became_active":"2024-01-27T03:45:08.074261+0000","last_became_peered":"2024-01-27T03:45:08.074261+0000","last_unstale":"2024-01-27T04:20:14.735085+0000","last_undegraded":"2024-01-27T04:20:14.735085+0000","last_fullsized":"2024-01-27T04:20:14.735085+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T15:20:03.504139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2560,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.733166+0000","last_change":"2024-01-27T03:45:29.625526+0000","last_active":"2024-01-27T04:20:16.733166+0000","last_peered":"2024-01-27T04:20:16.733166+0000","last_clean":"2024-01-27T04:20:16.733166+0000","last_became_active":"2024-01-27T03:45:29.620303+0000","last_became_peered":"2024-01-27T03:45:29.620303+0000","last_unstale":"2024-01-27T04:20:16.733166+0000","last_undegraded":"2024-01-27T04:20:16.733166+0000","last_fullsized":"2024-01-27T04:20:16.733166+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:09.679551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2554,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.737195+0000","last_change":"2024-01-27T03:45:11.908907+0000","last_active":"2024-01-27T04:20:16.737195+0000","last_peered":"2024-01-27T04:20:16.737195+0000","last_clean":"2024-01-27T04:20:16.737195+0000","last_became_active":"2024-01-27T03:45:11.902875+0000","last_became_peered":"2024-01-27T03:45:11.902875+0000","last_unstale":"2024-01-27T04:20:16.737195+0000","last_undegraded":"2024-01-27T04:20:16.737195+0000","last_fullsized":"2024-01-27T04:20:16.737195+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T14:33:16.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2547,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.831239+0000","last_change":"2024-01-27T03:45:41.083249+0000","last_active":"2024-01-27T04:20:13.831239+0000","last_peered":"2024-01-27T04:20:13.831239+0000","last_clean":"2024-01-27T04:20:13.831239+0000","last_became_active":"2024-01-27T03:45:41.077954+0000","last_became_peered":"2024-01-27T03:45:41.077954+0000","last_unstale":"2024-01-27T04:20:13.831239+0000","last_undegraded":"2024-01-27T04:20:13.831239+0000","last_fullsized":"2024-01-27T04:20:13.831239+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:53:19.777797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2555,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.730962+0000","last_change":"2024-01-27T03:45:13.508628+0000","last_active":"2024-01-27T04:20:16.730962+0000","last_peered":"2024-01-27T04:20:16.730962+0000","last_clean":"2024-01-27T04:20:16.730962+0000","last_became_active":"2024-01-27T03:45:13.503336+0000","last_became_peered":"2024-01-27T03:45:13.503336+0000","last_unstale":"2024-01-27T04:20:16.730962+0000","last_undegraded":"2024-01-27T04:20:16.730962+0000","last_fullsized":"2024-01-27T04:20:16.730962+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:08:42.711117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2551,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.291640+0000","last_change":"2024-01-27T03:45:06.421706+0000","last_active":"2024-01-27T04:20:12.291640+0000","last_peered":"2024-01-27T04:20:12.291640+0000","last_clean":"2024-01-27T04:20:12.291640+0000","last_became_active":"2024-01-27T03:45:06.412514+0000","last_became_peered":"2024-01-27T03:45:06.412514+0000","last_unstale":"2024-01-27T04:20:12.291640+0000","last_undegraded":"2024-01-27T04:20:12.291640+0000","last_fullsized":"2024-01-27T04:20:12.291640+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:23.739645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2562,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.337984+0000","last_change":"2024-01-27T03:45:05.911725+0000","last_active":"2024-01-27T04:20:12.337984+0000","last_peered":"2024-01-27T04:20:12.337984+0000","last_clean":"2024-01-27T04:20:12.337984+0000","last_became_active":"2024-01-27T03:45:05.905013+0000","last_became_peered":"2024-01-27T03:45:05.905013+0000","last_unstale":"2024-01-27T04:20:12.337984+0000","last_undegraded":"2024-01-27T04:20:12.337984+0000","last_fullsized":"2024-01-27T04:20:12.337984+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:23:34.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2552,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.277864+0000","last_change":"2024-01-27T03:45:09.267062+0000","last_active":"2024-01-27T04:20:12.277864+0000","last_peered":"2024-01-27T04:20:12.277864+0000","last_clean":"2024-01-27T04:20:12.277864+0000","last_became_active":"2024-01-27T03:45:08.513799+0000","last_became_peered":"2024-01-27T03:45:08.513799+0000","last_unstale":"2024-01-27T04:20:12.277864+0000","last_undegraded":"2024-01-27T04:20:12.277864+0000","last_fullsized":"2024-01-27T04:20:12.277864+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:45.278278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2571,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.448883+0000","last_change":"2024-01-27T03:45:12.810545+0000","last_active":"2024-01-27T04:20:16.448883+0000","last_peered":"2024-01-27T04:20:16.448883+0000","last_clean":"2024-01-27T04:20:16.448883+0000","last_became_active":"2024-01-27T03:45:12.797603+0000","last_became_peered":"2024-01-27T03:45:12.797603+0000","last_unstale":"2024-01-27T04:20:16.448883+0000","last_undegraded":"2024-01-27T04:20:16.448883+0000","last_fullsized":"2024-01-27T04:20:16.448883+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:06:00.673252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2532,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.622160+0000","last_change":"2024-01-27T03:45:08.152790+0000","last_active":"2024-01-27T04:20:15.622160+0000","last_peered":"2024-01-27T04:20:15.622160+0000","last_clean":"2024-01-27T04:20:15.622160+0000","last_became_active":"2024-01-27T03:45:08.146505+0000","last_became_peered":"2024-01-27T03:45:08.146505+0000","last_unstale":"2024-01-27T04:20:15.622160+0000","last_undegraded":"2024-01-27T04:20:15.622160+0000","last_fullsized":"2024-01-27T04:20:15.622160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:50.587416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2559,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.349162+0000","last_change":"2024-01-27T03:45:10.396512+0000","last_active":"2024-01-27T04:20:12.349162+0000","last_peered":"2024-01-27T04:20:12.349162+0000","last_clean":"2024-01-27T04:20:12.349162+0000","last_became_active":"2024-01-27T03:45:10.387717+0000","last_became_peered":"2024-01-27T03:45:10.387717+0000","last_unstale":"2024-01-27T04:20:12.349162+0000","last_undegraded":"2024-01-27T04:20:12.349162+0000","last_fullsized":"2024-01-27T04:20:12.349162+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:12:16.534326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2548,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.785854+0000","last_change":"2024-01-27T03:45:41.124551+0000","last_active":"2024-01-27T04:20:13.785854+0000","last_peered":"2024-01-27T04:20:13.785854+0000","last_clean":"2024-01-27T04:20:13.785854+0000","last_became_active":"2024-01-27T03:45:41.119306+0000","last_became_peered":"2024-01-27T03:45:41.119306+0000","last_unstale":"2024-01-27T04:20:13.785854+0000","last_undegraded":"2024-01-27T04:20:13.785854+0000","last_fullsized":"2024-01-27T04:20:13.785854+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:52:56.858068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2569,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.470531+0000","last_change":"2024-01-27T03:45:15.265604+0000","last_active":"2024-01-27T04:20:16.470531+0000","last_peered":"2024-01-27T04:20:16.470531+0000","last_clean":"2024-01-27T04:20:16.470531+0000","last_became_active":"2024-01-27T03:45:15.259193+0000","last_became_peered":"2024-01-27T03:45:15.259193+0000","last_unstale":"2024-01-27T04:20:16.470531+0000","last_undegraded":"2024-01-27T04:20:16.470531+0000","last_fullsized":"2024-01-27T04:20:16.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:48.948534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313662,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68372,"kb_used_data":5248,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303468,"statfs":{"total":96636764160,"available":96566751232,"internally_reserved":0,"allocated":5373952,"data_stored":2671367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":11.641999999999999,"5min":22.148,"15min":18.57},"min":{"1min":4.3710000000000004,"5min":4.3710000000000004,"15min":4.3710000000000004},"max":{"1min":35.859000000000002,"5min":232.63900000000001,"15min":232.63900000000001},"last":32.728999999999999},{"interface":"front","average":{"1min":15.961,"5min":23.016999999999999,"15min":19.32},"min":{"1min":4.8449999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"max":{"1min":48.558999999999997,"5min":265.815,"15min":265.815},"last":29.667999999999999}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.598000000000001,"5min":27.135000000000002,"15min":23.359999999999999},"min":{"1min":6.415,"5min":6.415,"15min":5.4749999999999996},"max":{"1min":55.116,"5min":152.09700000000001,"15min":421.90300000000002},"last":25.469999999999999},{"interface":"front","average":{"1min":12.83,"5min":26.446999999999999,"15min":21.591000000000001},"min":{"1min":6.3529999999999998,"5min":5.968,"15min":5.7039999999999997},"max":{"1min":40.463999999999999,"5min":165.53999999999999,"15min":319.13200000000001},"last":22.477}]},{"osd":3,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":16.913,"5min":26.349,"15min":24.771000000000001},"min":{"1min":7.9109999999999996,"5min":7.2750000000000004,"15min":6.0670000000000002},"max":{"1min":54.639000000000003,"5min":176.215,"15min":434.72800000000001},"last":37.106000000000002},{"interface":"front","average":{"1min":23.969000000000001,"5min":34.192999999999998,"15min":31.582999999999998},"min":{"1min":10.372999999999999,"5min":10.372999999999999,"15min":8.3019999999999996},"max":{"1min":95.001000000000005,"5min":216.80099999999999,"15min":447.06700000000001},"last":58.091999999999999}]},{"osd":4,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":19.324000000000002,"5min":24.849,"15min":23.100999999999999},"min":{"1min":8.2040000000000006,"5min":7.4240000000000004,"15min":7.0979999999999999},"max":{"1min":59.201000000000001,"5min":152.88,"15min":329.11500000000001},"last":38.07},{"interface":"front","average":{"1min":18.565999999999999,"5min":24.952999999999999,"15min":21.696999999999999},"min":{"1min":8.4870000000000001,"5min":7.1239999999999997,"15min":5.7839999999999998},"max":{"1min":61.326999999999998,"5min":233.845,"15min":233.845},"last":30.361999999999998}]},{"osd":5,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":25.951000000000001,"5min":30.747,"15min":26.004000000000001},"min":{"1min":8.7680000000000007,"5min":8.7680000000000007,"15min":7.165},"max":{"1min":101.246,"5min":252.73500000000001,"15min":324.52300000000002},"last":53.686},{"interface":"front","average":{"1min":27.847999999999999,"5min":38.902999999999999,"15min":30.933},"min":{"1min":13.746,"5min":11.076000000000001,"15min":9.0210000000000008},"max":{"1min":82.462999999999994,"5min":382,"15min":382},"last":35.701000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":29.972999999999999,"15min":28.385000000000002},"min":{"1min":7.335,"5min":7.335,"15min":7.335},"max":{"1min":33.107999999999997,"5min":165.976,"15min":400.93900000000002},"last":31.201000000000001},{"interface":"front","average":{"1min":16.268000000000001,"5min":28.919,"15min":27.251000000000001},"min":{"1min":9.4049999999999994,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":26.937999999999999,"5min":217.77699999999999,"15min":684.36699999999996},"last":34.392000000000003}]},{"osd":7,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":20.309999999999999,"5min":33.465000000000003,"15min":31.215},"min":{"1min":9.0519999999999996,"5min":8.1389999999999993,"15min":8.1389999999999993},"max":{"1min":50.585000000000001,"5min":242.08199999999999,"15min":443.21499999999997},"last":39.984000000000002},{"interface":"front","average":{"1min":27.695,"5min":32.735999999999997,"15min":28.863},"min":{"1min":12.496,"5min":9.6590000000000007,"15min":8.3309999999999995},"max":{"1min":133.63200000000001,"5min":145.751,"15min":436.80399999999997},"last":33.523000000000003}]}]},{"osd":2,"up_from":21,"seq":90194313660,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76292,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295548,"statfs":{"total":96636764160,"available":96558641152,"internally_reserved":0,"allocated":4972544,"data_stored":2273527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":8.7110000000000003,"5min":25.661999999999999,"15min":18.187999999999999},"min":{"1min":4.2939999999999996,"5min":4.2939999999999996,"15min":4.2939999999999996},"max":{"1min":23.433,"5min":445.62400000000002,"15min":445.62400000000002},"last":16.015999999999998},{"interface":"front","average":{"1min":12.178000000000001,"5min":34.600999999999999,"15min":22.532},"min":{"1min":7.1420000000000003,"5min":6.4370000000000003,"15min":5.3230000000000004},"max":{"1min":16.050000000000001,"5min":974.43299999999999,"15min":974.43299999999999},"last":21.056999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.199999999999999,"5min":37.404000000000003,"15min":25.739000000000001},"min":{"1min":7.0250000000000004,"5min":7.0250000000000004,"15min":6.6239999999999997},"max":{"1min":19.93,"5min":975.43499999999995,"15min":975.43499999999995},"last":8.5619999999999994},{"interface":"front","average":{"1min":12.119999999999999,"5min":28.672000000000001,"15min":22.556000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":5.5899999999999999},"max":{"1min":25.190000000000001,"5min":481.185,"15min":481.185},"last":12.433999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":17.236999999999998,"5min":36.584000000000003,"15min":26.488},"min":{"1min":7.665,"5min":7.2709999999999999,"15min":7.2709999999999999},"max":{"1min":59.902999999999999,"5min":980.83500000000004,"15min":980.83500000000004},"last":11.738},{"interface":"front","average":{"1min":18.442,"5min":41.698,"15min":29.129999999999999},"min":{"1min":11.558999999999999,"5min":10.58,"15min":9.5459999999999994},"max":{"1min":26.449000000000002,"5min":980.17499999999995,"15min":980.17499999999995},"last":15.047000000000001}]},{"osd":4,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":12.173999999999999,"5min":27.407,"15min":22.308},"min":{"1min":6.7809999999999997,"5min":6.2750000000000004,"15min":5.819},"max":{"1min":23.806000000000001,"5min":508.601,"15min":508.601},"last":8.2010000000000005},{"interface":"front","average":{"1min":14.736000000000001,"5min":34.642000000000003,"15min":27.102},"min":{"1min":9.7059999999999995,"5min":8.234,"15min":6.6580000000000004},"max":{"1min":27.920999999999999,"5min":492.29199999999997,"15min":492.29199999999997},"last":7.8460000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":18.317,"5min":48.280000000000001,"15min":33.753999999999998},"min":{"1min":8.9800000000000004,"5min":8.9800000000000004,"15min":8.9740000000000002},"max":{"1min":26.623999999999999,"5min":976.35900000000004,"15min":976.35900000000004},"last":17.640999999999998},{"interface":"front","average":{"1min":12.997,"5min":29.919,"15min":23.949000000000002},"min":{"1min":9.1140000000000008,"5min":6.9610000000000003,"15min":5.758},"max":{"1min":18.027999999999999,"5min":509.71699999999998,"15min":509.71699999999998},"last":10.263}]},{"osd":6,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":33.555999999999997,"15min":28.885000000000002},"min":{"1min":10.654999999999999,"5min":7.8869999999999996,"15min":6.3390000000000004},"max":{"1min":22.977,"5min":526.51400000000001,"15min":526.51400000000001},"last":14.087},{"interface":"front","average":{"1min":16.966000000000001,"5min":39.537999999999997,"15min":35.607999999999997},"min":{"1min":12.301,"5min":10.784000000000001,"15min":9.4350000000000005},"max":{"1min":23.710999999999999,"5min":533.20799999999997,"15min":533.20799999999997},"last":19.152000000000001}]},{"osd":7,"last update":"Sat Jan 27 04:19:26 2024","interfaces":[{"interface":"back","average":{"1min":25.797999999999998,"5min":49.715000000000003,"15min":36.329000000000001},"min":{"1min":14.427,"5min":11.739000000000001,"15min":9.5139999999999993},"max":{"1min":74.819000000000003,"5min":977.13800000000003,"15min":977.13800000000003},"last":20.050999999999998},{"interface":"front","average":{"1min":18.117000000000001,"5min":33.683,"15min":29.026},"min":{"1min":11.587999999999999,"5min":8.5850000000000009,"15min":8.5850000000000009},"max":{"1min":52.927999999999997,"5min":549.30100000000004,"15min":549.30100000000004},"last":16.943999999999999}]}]},{"osd":6,"up_from":20,"seq":85899346368,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72180,"kb_used_data":4896,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299660,"statfs":{"total":96636764160,"available":96562851840,"internally_reserved":0,"allocated":5013504,"data_stored":2295222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":9.9730000000000008,"5min":22.446000000000002,"15min":18.524999999999999},"min":{"1min":5.492,"5min":4.6929999999999996,"15min":4.2350000000000003},"max":{"1min":20.303000000000001,"5min":339.68000000000001,"15min":339.68000000000001},"last":9.9979999999999993},{"interface":"front","average":{"1min":14.538,"5min":32.518000000000001,"15min":26.594999999999999},"min":{"1min":9.9329999999999998,"5min":5.5049999999999999,"15min":4.9749999999999996},"max":{"1min":27.638999999999999,"5min":341.03800000000001,"15min":341.03800000000001},"last":21.902999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":12.906000000000001,"5min":24.239999999999998,"15min":18.077999999999999},"min":{"1min":6.1100000000000003,"5min":5.1970000000000001,"15min":4.6950000000000003},"max":{"1min":35.095999999999997,"5min":250.96000000000001,"15min":250.96000000000001},"last":6.9530000000000003},{"interface":"front","average":{"1min":12.737,"5min":29.638999999999999,"15min":23.449000000000002},"min":{"1min":6.181,"5min":5.0679999999999996,"15min":4.7670000000000003},"max":{"1min":27.792000000000002,"5min":350.74599999999998,"15min":350.74599999999998},"last":8.3989999999999991}]},{"osd":2,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":12.606999999999999,"5min":31.344999999999999,"15min":23.207000000000001},"min":{"1min":6.6669999999999998,"5min":6.6669999999999998,"15min":6.0839999999999996},"max":{"1min":25.248999999999999,"5min":349.19299999999998,"15min":349.19299999999998},"last":18.024000000000001},{"interface":"front","average":{"1min":14.715,"5min":35.548000000000002,"15min":29.663},"min":{"1min":9.3729999999999993,"5min":6.5499999999999998,"15min":6.4630000000000001},"max":{"1min":23.071999999999999,"5min":354.80799999999999,"15min":354.80799999999999},"last":24.498999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:43 2024","interfaces":[{"interface":"back","average":{"1min":16.457000000000001,"5min":32.703000000000003,"15min":24.315000000000001},"min":{"1min":6.9299999999999997,"5min":6.9279999999999999,"15min":6.7469999999999999},"max":{"1min":54.694000000000003,"5min":247.446,"15min":265.50299999999999},"last":9.25},{"interface":"front","average":{"1min":20.056999999999999,"5min":34.040999999999997,"15min":28.632999999999999},"min":{"1min":10.635999999999999,"5min":6.4930000000000003,"15min":6.4930000000000003},"max":{"1min":49.180999999999997,"5min":341.68400000000003,"15min":352.79399999999998},"last":26.282}]},{"osd":4,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":15.965,"5min":27.364999999999998,"15min":24.411000000000001},"min":{"1min":10.279999999999999,"5min":6.7590000000000003,"15min":6.5519999999999996},"max":{"1min":33.625999999999998,"5min":243.96100000000001,"15min":266.12400000000002},"last":20.373000000000001},{"interface":"front","average":{"1min":14.163,"5min":30.145,"15min":27.082000000000001},"min":{"1min":7.359,"5min":6.6859999999999999,"15min":6.0289999999999999},"max":{"1min":29.823,"5min":348.11500000000001,"15min":348.11500000000001},"last":18.652999999999999}]},{"osd":5,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":17.280999999999999,"5min":30.981999999999999,"15min":26.800000000000001},"min":{"1min":11.436999999999999,"5min":7.827,"15min":7.827},"max":{"1min":28.574000000000002,"5min":295.56,"15min":295.56},"last":28.443999999999999},{"interface":"front","average":{"1min":19.364000000000001,"5min":37.106000000000002,"15min":30.992000000000001},"min":{"1min":11.782,"5min":10.959,"15min":8.3569999999999993},"max":{"1min":47.966999999999999,"5min":349.81,"15min":354.55099999999999},"last":29.861999999999998}]},{"osd":7,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":26.315000000000001,"5min":38.106000000000002,"15min":25.777000000000001},"min":{"1min":14.762,"5min":9.7609999999999992,"15min":7.4199999999999999},"max":{"1min":93.060000000000002,"5min":346.697,"15min":346.697},"last":26.91},{"interface":"front","average":{"1min":18.989999999999998,"5min":37.228000000000002,"15min":28.795000000000002},"min":{"1min":11.340999999999999,"5min":7.7670000000000003,"15min":7.7670000000000003},"max":{"1min":49.277000000000001,"5min":352.11900000000003,"15min":369.92899999999997},"last":19.280000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346365,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72164,"kb_used_data":4888,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299676,"statfs":{"total":96636764160,"available":96562868224,"internally_reserved":0,"allocated":5005312,"data_stored":2288496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":12.260999999999999,"5min":21.777000000000001,"15min":17.795999999999999},"min":{"1min":4.5209999999999999,"5min":4.4020000000000001,"15min":3.782},"max":{"1min":30.248000000000001,"5min":271.60599999999999,"15min":271.60599999999999},"last":30.248000000000001},{"interface":"front","average":{"1min":11.218999999999999,"5min":19.111999999999998,"15min":15.164999999999999},"min":{"1min":4.8700000000000001,"5min":4.8700000000000001,"15min":4.7830000000000004},"max":{"1min":36.665999999999997,"5min":141.624,"15min":141.624},"last":22.373999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":15.82,"5min":20.286999999999999,"15min":20.946000000000002},"min":{"1min":7.3730000000000002,"5min":5.8159999999999998,"15min":5.8159999999999998},"max":{"1min":44.912999999999997,"5min":228.27600000000001,"15min":228.27600000000001},"last":32.338000000000001},{"interface":"front","average":{"1min":13.35,"5min":23.087,"15min":21.771999999999998},"min":{"1min":7.7060000000000004,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":41.997,"5min":276.93299999999999,"15min":276.93299999999999},"last":28.045000000000002}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":18.422999999999998,"5min":21.227,"15min":22.727},"min":{"1min":7.9669999999999996,"5min":7.8319999999999999,"15min":6.4219999999999997},"max":{"1min":62.054000000000002,"5min":138.13300000000001,"15min":252.38399999999999},"last":30.907},{"interface":"front","average":{"1min":16.931999999999999,"5min":26.645,"15min":22.036999999999999},"min":{"1min":9.7780000000000005,"5min":7.4930000000000003,"15min":5.8979999999999997},"max":{"1min":48.671999999999997,"5min":305.358,"15min":321.09399999999999},"last":23.082999999999998}]},{"osd":3,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":19.376000000000001,"5min":22.815999999999999,"15min":22.629999999999999},"min":{"1min":10.507999999999999,"5min":8.452,"15min":8.1780000000000008},"max":{"1min":61.652999999999999,"5min":89.129999999999995,"15min":195.47800000000001},"last":21.466999999999999},{"interface":"front","average":{"1min":14.627000000000001,"5min":17.146999999999998,"15min":16.039999999999999},"min":{"1min":6.4699999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":91.978999999999999,"5min":120.044,"15min":154.279},"last":7.0910000000000002}]},{"osd":5,"last update":"Sat Jan 27 04:19:38 2024","interfaces":[{"interface":"back","average":{"1min":16.870999999999999,"5min":26.902000000000001,"15min":21.398},"min":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.1779999999999999},"max":{"1min":57.954999999999998,"5min":243.983,"15min":243.983},"last":43.405999999999999},{"interface":"front","average":{"1min":21.181000000000001,"5min":27.506,"15min":24.664000000000001},"min":{"1min":11.605,"5min":9.7690000000000001,"15min":9.7200000000000006},"max":{"1min":61.289000000000001,"5min":188.453,"15min":191.036},"last":59.655000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":21.027000000000001,"5min":29.692,"15min":27.587},"min":{"1min":8.3409999999999993,"5min":8.3409999999999993,"15min":8.3409999999999993},"max":{"1min":46.170999999999999,"5min":259.351,"15min":259.351},"last":8.3409999999999993},{"interface":"front","average":{"1min":24.855,"5min":30.391999999999999,"15min":24.253},"min":{"1min":14.805,"5min":6.9089999999999998,"15min":6.4630000000000001},"max":{"1min":49.533999999999999,"5min":295.00900000000001,"15min":295.00900000000001},"last":25.052}]},{"osd":7,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":20.744,"5min":25.904,"15min":21.736999999999998},"min":{"1min":10.656000000000001,"5min":8.1850000000000005,"15min":6.4269999999999996},"max":{"1min":52.637999999999998,"5min":206.73400000000001,"15min":206.73400000000001},"last":9.5030000000000001},{"interface":"front","average":{"1min":26.753,"5min":32.119,"15min":27.643000000000001},"min":{"1min":15.661,"5min":10.401999999999999,"15min":9.7940000000000005},"max":{"1min":90.658000000000001,"5min":262.28300000000002,"15min":262.28300000000002},"last":29.603999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477181,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72164,"kb_used_data":4884,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299676,"statfs":{"total":96636764160,"available":96562868224,"internally_reserved":0,"allocated":5001216,"data_stored":2288491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:36 2024","interfaces":[{"interface":"back","average":{"1min":11.134,"5min":16.122,"15min":16.141999999999999},"min":{"1min":5.3609999999999998,"5min":5.3609999999999998,"15min":4.6310000000000002},"max":{"1min":33.012,"5min":93.953000000000003,"15min":238.642},"last":13.462},{"interface":"front","average":{"1min":14.760999999999999,"5min":20.975000000000001,"15min":20.623999999999999},"min":{"1min":8.0269999999999992,"5min":7.274,"15min":7.274},"max":{"1min":34.912999999999997,"5min":135.18299999999999,"15min":230.04900000000001},"last":14.769}]},{"osd":1,"last update":"Sat Jan 27 04:19:16 2024","interfaces":[{"interface":"back","average":{"1min":10.414,"5min":15.129,"15min":15.5},"min":{"1min":6.0369999999999999,"5min":5.0730000000000004,"15min":4.7359999999999998},"max":{"1min":27.911000000000001,"5min":103.377,"15min":188.899},"last":6.1779999999999999},{"interface":"front","average":{"1min":10.236000000000001,"5min":16.545000000000002,"15min":20.106000000000002},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":30.829000000000001,"5min":102.009,"15min":218.97399999999999},"last":18.475999999999999}]},{"osd":2,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":16.824000000000002,"5min":22.969999999999999,"15min":21.879999999999999},"min":{"1min":7.8319999999999999,"5min":7.8319999999999999,"15min":7.8319999999999999},"max":{"1min":44.707000000000001,"5min":130.16499999999999,"15min":322.24599999999998},"last":16.512},{"interface":"front","average":{"1min":19.797999999999998,"5min":25.126000000000001,"15min":25.841999999999999},"min":{"1min":11,"5min":9.8699999999999992,"15min":9.8699999999999992},"max":{"1min":53.359000000000002,"5min":201.851,"15min":326.04700000000003},"last":20.082999999999998}]},{"osd":4,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":22.923999999999999,"5min":27.471,"15min":25.827000000000002},"min":{"1min":11.742000000000001,"5min":10.131,"15min":10.131},"max":{"1min":115.759,"5min":220.51300000000001,"15min":250.255},"last":19.363},{"interface":"front","average":{"1min":19.719999999999999,"5min":20.579999999999998,"15min":17.524999999999999},"min":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.5049999999999999},"max":{"1min":150.36199999999999,"5min":221.44999999999999,"15min":227.67400000000001},"last":7.9390000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:19:29 2024","interfaces":[{"interface":"back","average":{"1min":15.736000000000001,"5min":17.591000000000001,"15min":15.472},"min":{"1min":7.2169999999999996,"5min":5.5979999999999999,"15min":4.9359999999999999},"max":{"1min":49.698999999999998,"5min":212.291,"15min":212.291},"last":17.169},{"interface":"front","average":{"1min":16.992999999999999,"5min":19.856999999999999,"15min":19.529},"min":{"1min":8.782,"5min":6.742,"15min":6.3550000000000004},"max":{"1min":70.760000000000005,"5min":147.773,"15min":237.16300000000001},"last":15.65}]},{"osd":6,"last update":"Sat Jan 27 04:19:57 2024","interfaces":[{"interface":"back","average":{"1min":15.177,"5min":20.619,"15min":19.541},"min":{"1min":8.1270000000000007,"5min":7.4939999999999998,"15min":6.1760000000000002},"max":{"1min":27.283999999999999,"5min":145.55199999999999,"15min":261.56799999999998},"last":12.894},{"interface":"front","average":{"1min":16.452000000000002,"5min":23.617999999999999,"15min":23.219999999999999},"min":{"1min":11.131,"5min":7.8700000000000001,"15min":7.8700000000000001},"max":{"1min":34.841999999999999,"5min":135.69900000000001,"15min":276.86099999999999},"last":10.686}]},{"osd":7,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":27.474,"5min":29.882999999999999,"15min":26.277999999999999},"min":{"1min":12.359999999999999,"5min":8.0449999999999999,"15min":8.0449999999999999},"max":{"1min":188.46100000000001,"5min":188.46100000000001,"15min":382.57999999999998},"last":22.079000000000001},{"interface":"front","average":{"1min":24.861000000000001,"5min":28.331,"15min":22.138999999999999},"min":{"1min":9.2249999999999996,"5min":9.2249999999999996,"15min":8.9849999999999994},"max":{"1min":222.232,"5min":222.232,"15min":254.12},"last":10.050000000000001}]}]},{"osd":0,"up_from":16,"seq":68719477179,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72160,"kb_used_data":4884,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299680,"statfs":{"total":96636764160,"available":96562872320,"internally_reserved":0,"allocated":5001216,"data_stored":2288491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":11.346,"5min":21.847999999999999,"15min":19.535},"min":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"max":{"1min":28.620000000000001,"5min":197.15600000000001,"15min":286.24799999999999},"last":9.8729999999999993},{"interface":"front","average":{"1min":13.196999999999999,"5min":29.672999999999998,"15min":23.280999999999999},"min":{"1min":6.726,"5min":6.0350000000000001,"15min":4.7960000000000003},"max":{"1min":36.061,"5min":254.524,"15min":267.99299999999999},"last":8.3680000000000003}]},{"osd":2,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":12.795999999999999,"5min":24.466999999999999,"15min":21.399000000000001},"min":{"1min":6.4829999999999997,"5min":5.984,"15min":5.1449999999999996},"max":{"1min":19.736000000000001,"5min":229.99299999999999,"15min":330.28699999999998},"last":15.855},{"interface":"front","average":{"1min":16.109999999999999,"5min":28.257000000000001,"15min":22.437999999999999},"min":{"1min":10.855,"5min":7.6020000000000003,"15min":6.1669999999999998},"max":{"1min":30.111999999999998,"5min":218.97900000000001,"15min":289.72899999999998},"last":8.9809999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:27 2024","interfaces":[{"interface":"back","average":{"1min":12.762,"5min":23.349,"15min":19.062999999999999},"min":{"1min":6.3230000000000004,"5min":6.3230000000000004,"15min":6.1840000000000002},"max":{"1min":28.111999999999998,"5min":156.90100000000001,"15min":243.779},"last":7.4039999999999999},{"interface":"front","average":{"1min":19.774999999999999,"5min":31.922999999999998,"15min":25.346},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.0860000000000003},"max":{"1min":32.655999999999999,"5min":265.04300000000001,"15min":265.04300000000001},"last":14.35}]},{"osd":4,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":22.652000000000001,"5min":31.033000000000001,"15min":24.866},"min":{"1min":8.2270000000000003,"5min":7.4429999999999996,"15min":7.1870000000000003},"max":{"1min":90.774000000000001,"5min":238.68600000000001,"15min":238.68600000000001},"last":10.619999999999999},{"interface":"front","average":{"1min":26.318999999999999,"5min":30.364999999999998,"15min":27.841000000000001},"min":{"1min":10.281000000000001,"5min":9.7170000000000005,"15min":9.5559999999999992},"max":{"1min":89.200000000000003,"5min":174.03800000000001,"15min":242.22800000000001},"last":16.584}]},{"osd":5,"last update":"Sat Jan 27 04:19:27 2024","interfaces":[{"interface":"back","average":{"1min":17.667999999999999,"5min":40.302999999999997,"15min":26.126999999999999},"min":{"1min":10.754,"5min":7.0670000000000002,"15min":6.665},"max":{"1min":34.997,"5min":385.50799999999998,"15min":385.50799999999998},"last":19.373999999999999},{"interface":"front","average":{"1min":21.75,"5min":42.832999999999998,"15min":28.548999999999999},"min":{"1min":11.871,"5min":9.6639999999999997,"15min":6.5839999999999996},"max":{"1min":47.709000000000003,"5min":337.73700000000002,"15min":337.73700000000002},"last":17.199000000000002}]},{"osd":6,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.641999999999999,"5min":39.985999999999997,"15min":31.425000000000001},"min":{"1min":13.096,"5min":10.368,"15min":10.368},"max":{"1min":36.189999999999998,"5min":236.387,"15min":248.78899999999999},"last":11.795},{"interface":"front","average":{"1min":17.960999999999999,"5min":34.942,"15min":28.195},"min":{"1min":7.9359999999999999,"5min":6.4889999999999999,"15min":6.4889999999999999},"max":{"1min":88.625,"5min":252.99700000000001,"15min":381.90800000000002},"last":14.997999999999999}]},{"osd":7,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.481000000000002,"5min":32.908000000000001,"15min":28.715},"min":{"1min":8.6880000000000006,"5min":8.6880000000000006,"15min":8.4670000000000005},"max":{"1min":56.813000000000002,"5min":232.21600000000001,"15min":236.81299999999999},"last":12.441000000000001},{"interface":"front","average":{"1min":21.489999999999998,"5min":37.271000000000001,"15min":28.934000000000001},"min":{"1min":12.420999999999999,"5min":10.638,"15min":7.2409999999999997},"max":{"1min":34.862000000000002,"5min":249.29499999999999,"15min":249.29499999999999},"last":13.699999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509882,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72540,"kb_used_data":5264,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299300,"statfs":{"total":96636764160,"available":96562483200,"internally_reserved":0,"allocated":5390336,"data_stored":2682863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":21.77,"5min":23.620999999999999,"15min":18.902000000000001},"min":{"1min":5.0140000000000002,"5min":4.4569999999999999,"15min":4.3319999999999999},"max":{"1min":203.28899999999999,"5min":232.74199999999999,"15min":278.56900000000002},"last":5.2510000000000003},{"interface":"front","average":{"1min":19.181999999999999,"5min":22.715,"15min":17.931000000000001},"min":{"1min":7.4649999999999999,"5min":5.4530000000000003,"15min":4.8129999999999997},"max":{"1min":115.354,"5min":234.15799999999999,"15min":305.03800000000001},"last":9.2330000000000005}]},{"osd":1,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":31.167000000000002,"5min":26.742000000000001,"15min":23.835999999999999},"min":{"1min":7.3579999999999997,"5min":6.0549999999999997,"15min":6.0549999999999997},"max":{"1min":192.631,"5min":227.602,"15min":304.28300000000002},"last":14.456},{"interface":"front","average":{"1min":28.222000000000001,"5min":23.238,"15min":20.527000000000001},"min":{"1min":6.5300000000000002,"5min":6.5300000000000002,"15min":5.0199999999999996},"max":{"1min":175.59700000000001,"5min":175.59700000000001,"15min":224.27199999999999},"last":8.3580000000000005}]},{"osd":2,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":23.504999999999999,"5min":30.835000000000001,"15min":29.007000000000001},"min":{"1min":6.375,"5min":6.375,"15min":6.375},"max":{"1min":105.164,"5min":239.91900000000001,"15min":311.17200000000003},"last":17.838000000000001},{"interface":"front","average":{"1min":22.550000000000001,"5min":26.795000000000002,"15min":25.391999999999999},"min":{"1min":7.056,"5min":6.8170000000000002,"15min":6.8170000000000002},"max":{"1min":104.66500000000001,"5min":209.00899999999999,"15min":314.78699999999998},"last":11.32}]},{"osd":3,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":26.381,"5min":25.306000000000001,"15min":20.504000000000001},"min":{"1min":7.7729999999999997,"5min":6.7690000000000001,"15min":6.274},"max":{"1min":179.31299999999999,"5min":187.23099999999999,"15min":274.71899999999999},"last":13.635},{"interface":"front","average":{"1min":24.838000000000001,"5min":28.155999999999999,"15min":24.609999999999999},"min":{"1min":11.1,"5min":8.0670000000000002,"15min":7.3819999999999997},"max":{"1min":108.70099999999999,"5min":175.44800000000001,"15min":233.72800000000001},"last":12.835000000000001}]},{"osd":4,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":24.469000000000001,"5min":26.721,"15min":22.181000000000001},"min":{"1min":8.5410000000000004,"5min":6.4269999999999996,"15min":5.9059999999999997},"max":{"1min":132.90299999999999,"5min":185.755,"15min":219.21899999999999},"last":9.9589999999999996},{"interface":"front","average":{"1min":26.553000000000001,"5min":30.170999999999999,"15min":25.722000000000001},"min":{"1min":9.5739999999999998,"5min":8.7789999999999999,"15min":7.6689999999999996},"max":{"1min":116.852,"5min":174.81,"15min":174.81},"last":15.851000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":25.891999999999999,"5min":31.309000000000001,"15min":27.731999999999999},"min":{"1min":10.345000000000001,"5min":6.9299999999999997,"15min":6.9299999999999997},"max":{"1min":96.703999999999994,"5min":234.95099999999999,"15min":259.803},"last":18.640999999999998},{"interface":"front","average":{"1min":30.484000000000002,"5min":33.567999999999998,"15min":29.574999999999999},"min":{"1min":10.73,"5min":10.73,"15min":8.2859999999999996},"max":{"1min":98.491,"5min":206.55799999999999,"15min":206.55799999999999},"last":17.457999999999998}]},{"osd":7,"last update":"Sat Jan 27 04:20:14 2024","interfaces":[{"interface":"back","average":{"1min":33.548999999999999,"5min":31.448,"15min":26.803000000000001},"min":{"1min":11.276999999999999,"5min":8.0739999999999998,"15min":8.0739999999999998},"max":{"1min":180.07300000000001,"5min":190.39099999999999,"15min":319.75400000000002},"last":20.158999999999999},{"interface":"front","average":{"1min":32.110999999999997,"5min":34.119999999999997,"15min":28.417999999999999},"min":{"1min":10.561999999999999,"5min":9.0800000000000001,"15min":6.6459999999999999},"max":{"1min":217.697,"5min":240.69800000000001,"15min":277.59500000000003},"last":15.109}]}]},{"osd":7,"up_from":13,"seq":55834575291,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72164,"kb_used_data":4888,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299676,"statfs":{"total":96636764160,"available":96562868224,"internally_reserved":0,"allocated":5005312,"data_stored":2288496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:00 2024","interfaces":[{"interface":"back","average":{"1min":10.483000000000001,"5min":21.335000000000001,"15min":15.542999999999999},"min":{"1min":5.649,"5min":4.7370000000000001,"15min":4.1779999999999999},"max":{"1min":24.736999999999998,"5min":162.80799999999999,"15min":162.80799999999999},"last":6.8440000000000003},{"interface":"front","average":{"1min":13.324,"5min":28.123999999999999,"15min":19.064},"min":{"1min":6.5460000000000003,"5min":4.4020000000000001,"15min":4.4020000000000001},"max":{"1min":22.442,"5min":265.62200000000001,"15min":265.62200000000001},"last":11.18}]},{"osd":1,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":14.460000000000001,"5min":25.378,"15min":21.184000000000001},"min":{"1min":9.5470000000000006,"5min":6.0899999999999999,"15min":6.0899999999999999},"max":{"1min":22.472999999999999,"5min":171.16800000000001,"15min":204.17099999999999},"last":11.798999999999999},{"interface":"front","average":{"1min":8.282,"5min":20.821000000000002,"15min":18.806999999999999},"min":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"max":{"1min":20.916,"5min":192.11000000000001,"15min":323.48599999999999},"last":14.613}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.647,"5min":30.683,"15min":21.582999999999998},"min":{"1min":9.0679999999999996,"5min":6.1779999999999999,"15min":6.1269999999999998},"max":{"1min":23.294,"5min":420.65199999999999,"15min":420.65199999999999},"last":9.3480000000000008},{"interface":"front","average":{"1min":13.297000000000001,"5min":28.969000000000001,"15min":22.788},"min":{"1min":6.25,"5min":6.25,"15min":6.25},"max":{"1min":36.289000000000001,"5min":453.39100000000002,"15min":453.39100000000002},"last":13.087999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":14.026,"5min":27.434000000000001,"15min":23.035},"min":{"1min":9.8599999999999994,"5min":8.0830000000000002,"15min":7.6890000000000001},"max":{"1min":23.341999999999999,"5min":226.72300000000001,"15min":226.72300000000001},"last":8.4209999999999994},{"interface":"front","average":{"1min":15.734999999999999,"5min":24.466000000000001,"15min":21.619},"min":{"1min":7.6680000000000001,"5min":5.7270000000000003,"15min":5.7270000000000003},"max":{"1min":23.318999999999999,"5min":197.01499999999999,"15min":197.01499999999999},"last":12.484}]},{"osd":4,"last update":"Sat Jan 27 04:20:11 2024","interfaces":[{"interface":"back","average":{"1min":15.734,"5min":29.295000000000002,"15min":22.515000000000001},"min":{"1min":9.2780000000000005,"5min":8.4749999999999996,"15min":8.3200000000000003},"max":{"1min":48.780000000000001,"5min":227.72300000000001,"15min":227.72300000000001},"last":10.061999999999999},{"interface":"front","average":{"1min":18.984000000000002,"5min":32.296999999999997,"15min":24.923999999999999},"min":{"1min":11.102,"5min":9.3059999999999992,"15min":9.3059999999999992},"max":{"1min":55.207000000000001,"5min":230.16,"15min":230.16},"last":18.850000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:19:22 2024","interfaces":[{"interface":"back","average":{"1min":18.823,"5min":38.701000000000001,"15min":28.065000000000001},"min":{"1min":14.417,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":49.390999999999998,"5min":276.23500000000001,"15min":276.23500000000001},"last":13.885},{"interface":"front","average":{"1min":21.015999999999998,"5min":35.655000000000001,"15min":29.832999999999998},"min":{"1min":16.212,"5min":10.522,"15min":10.522},"max":{"1min":36.661000000000001,"5min":408.774,"15min":408.774},"last":18.050000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":20.045999999999999,"5min":34.045000000000002,"15min":29.82},"min":{"1min":15.138,"5min":10.936,"15min":10.936},"max":{"1min":27.131,"5min":276.92099999999999,"15min":304.952},"last":17.254000000000001},{"interface":"front","average":{"1min":14.42,"5min":30.312000000000001,"15min":26.094000000000001},"min":{"1min":7.2199999999999998,"5min":7.2199999999999998,"15min":7.2199999999999998},"max":{"1min":25.401,"5min":302.61700000000002,"15min":302.61700000000002},"last":15.374000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T04:20:18.770 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-27T04:20:18.770 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T04:20:18.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.907+0000 7fce84f2c640 10 monclient: get_monmap_and_config 2024-01-27T04:20:18.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.907+0000 7fce84f2c640 10 monclient: build_initial_monmap 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.907+0000 7fce84f2c640 10 monclient: monmap: 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:18.908225+0000 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:18.908225+0000 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:18.909 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.907+0000 7fce84f2c640 1 Processor -- start 2024-01-27T04:20:18.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.907+0000 7fce84f2c640 1 -- start start 2024-01-27T04:20:18.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.907+0000 7fce84f2c640 10 monclient: init 2024-01-27T04:20:18.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce800b4680 0x7fce80142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 10 monclient(hunting): picked mon.noname-b con 0x7fce800b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce800b4a60 0x7fce800b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 10 monclient(hunting): picked mon.noname-a con 0x7fce800b4a60 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce80142eb0 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce800afbe0 con 0x7fce80142eb0 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce80142240 con 0x7fce800b4a60 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce800b0890 con 0x7fce800b4680 2024-01-27T04:20:18.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce84f2c640 10 monclient(hunting): authenticate will time out at 3702.262385s 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.908+0000 7fce7e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce800b4680 0x7fce80142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.909+0000 7fce7e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce800b4680 0x7fce80142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:41110/0 (socket says 172.21.15.110:41110) 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.909+0000 7fce7e59c640 1 -- 172.21.15.110:0/3967458437 learned_addr learned my addr 172.21.15.110:0/3967458437 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.909+0000 7fce7e59c640 10 monclient(hunting): get_auth_request con 0x7fce800b4680 auth_method 0 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.909+0000 7fce7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.909+0000 7fce7e59c640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.909+0000 7fce7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.909+0000 7fce7e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.909+0000 7fce7e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.909+0000 7fce7e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:18.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce77fff640 1 --2- 172.21.15.110:0/3967458437 >> v2:172.21.15.110:3300/0 conn(0x7fce800b4a60 0x7fce800b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce7e59c640 10 monclient(hunting): handle_auth_done global_id 18857 payload 274 2024-01-27T04:20:18.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce7e59c640 1 -- 172.21.15.110:0/3967458437 >> v2:172.21.15.110:3300/0 conn(0x7fce800b4a60 msgr2=0x7fce800b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:18.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce7e59c640 1 --2- 172.21.15.110:0/3967458437 >> v2:172.21.15.110:3300/0 conn(0x7fce800b4a60 0x7fce800b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce7e59c640 1 -- 172.21.15.110:0/3967458437 >> v1:172.21.15.110:6789/0 conn(0x7fce80142eb0 legacy=0x7fce800b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:18.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce7e59c640 10 monclient: _finish_hunting 0 2024-01-27T04:20:18.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce7e59c640 1 monclient: found mon.noname-b 2024-01-27T04:20:18.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:18.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce7e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:18.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.911+0000 7fce7e59c640 1 -- 172.21.15.110:0/3967458437 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce800b0b00 con 0x7fce800b4680 2024-01-27T04:20:18.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.913+0000 7fce7e59c640 1 --2- 172.21.15.110:0/3967458437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce800b4680 0x7fce80142970 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7fce70009870 tx=0x7fce7002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=196713ed96fea6dc server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:18.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.915+0000 7fce7dd9b640 1 -- 172.21.15.110:0/3967458437 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce700317c0 con 0x7fce800b4680 2024-01-27T04:20:18.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.915+0000 7fce7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:18.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.915+0000 7fce7dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.915+0000 7fce7dd9b640 10 monclient: dump: 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.915+0000 7fce7dd9b640 10 monclient: _finish_auth 0 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.915+0000 7fce7dd9b640 10 monclient: _check_auth_tickets 2024-01-27T04:20:18.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.915+0000 7fce7dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:18.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.915+0000 7fce84f2c640 5 monclient: authenticate success, global_id 18857 2024-01-27T04:20:18.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.915+0000 7fce84f2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:18.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.917+0000 7fce7dd9b640 1 -- 172.21.15.110:0/3967458437 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce70031920 con 0x7fce800b4680 2024-01-27T04:20:18.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.917+0000 7fce7dd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:18.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.917+0000 7fce7dd9b640 1 -- 172.21.15.110:0/3967458437 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce70031cb0 con 0x7fce800b4680 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.917+0000 7fce84f2c640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.917+0000 7fce84f2c640 4 set_mon_vals no callback set 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.917+0000 7fce7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.917+0000 7fce7dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.917+0000 7fce7dd9b640 10 monclient: dump: 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:18.919 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 10 monclient: shutdown 2024-01-27T04:20:18.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:18.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 1 -- 172.21.15.110:0/3967458437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce800b4680 msgr2=0x7fce80142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:18.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 1 --2- 172.21.15.110:0/3967458437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce800b4680 0x7fce80142970 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7fce70009870 tx=0x7fce7002b860 comp rx=0 tx=0).stop 2024-01-27T04:20:18.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 1 -- 172.21.15.110:0/3967458437 shutdown_connections 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 1 --2- 172.21.15.110:0/3967458437 >> v2:172.21.15.110:3300/0 conn(0x7fce800b4a60 0x7fce800b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 1 --2- 172.21.15.110:0/3967458437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce800b4680 0x7fce80142970 unknown :-1 s=CLOSED pgs=3406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 1 -- 172.21.15.110:0/3967458437 >> 172.21.15.110:0/3967458437 conn(0x7fce80066490 msgr2=0x7fce800668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 1 -- 172.21.15.110:0/3967458437 shutdown_connections 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.918+0000 7fce84f2c640 1 -- 172.21.15.110:0/3967458437 wait complete. 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.919+0000 7fce84f2c640 10 monclient: build_initial_monmap 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.919+0000 7fce84f2c640 10 monclient: monmap: 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.923 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.919+0000 7fce84f2c640 1 librados: starting msgr at 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.919+0000 7fce84f2c640 1 librados: starting objecter 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.919+0000 7fce84f2c640 1 Processor -- start 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.919+0000 7fce84f2c640 1 -- start start 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.919+0000 7fce84f2c640 1 librados: setting wanted keys 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.919+0000 7fce84f2c640 1 librados: calling monclient init 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.919+0000 7fce84f2c640 10 monclient: init 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce800b4680 0x7fce8013ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient(hunting): picked mon.noname-a con 0x7fce800b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce80142eb0 0x7fce800aac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient(hunting): picked mon.noname-c con 0x7fce80142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce800ab810 0x7fce800abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient(hunting): picked mon.noname-b con 0x7fce800ab810 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce800acfb0 con 0x7fce800b4680 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce800ad290 con 0x7fce800ab810 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce800ab2c0 con 0x7fce80142eb0 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce84f2c640 10 monclient(hunting): authenticate will time out at 3702.274246s 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.920+0000 7fce77fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce80142eb0 0x7fce800aac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce7ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce800ab810 0x7fce800abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce77fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce80142eb0 0x7fce800aac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:41114/0 (socket says 172.21.15.110:41114) 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce77fff640 1 -- 172.21.15.110:0/3210708782 learned_addr learned my addr 172.21.15.110:0/3210708782 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce77fff640 10 monclient(hunting): get_auth_request con 0x7fce80142eb0 auth_method 0 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce77fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce77fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce7ed9d640 10 monclient(hunting): get_auth_request con 0x7fce800ab810 auth_method 0 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce7ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.921+0000 7fce7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.922+0000 7fce7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.922+0000 7fce7e59c640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce800b4680 0x7fce8013ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.923+0000 7fce7e59c640 10 monclient(hunting): get_auth_request con 0x7fce800b4680 auth_method 0 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.923+0000 7fce7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.923+0000 7fce7e59c640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.923+0000 7fce7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 10 monclient(hunting): handle_auth_done global_id 18860 payload 824 2024-01-27T04:20:18.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 1 -- 172.21.15.110:0/3210708782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce800ab810 msgr2=0x7fce800abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce800ab810 0x7fce800abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 1 -- 172.21.15.110:0/3210708782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce800b4680 msgr2=0x7fce8013ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce800b4680 0x7fce8013ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 10 monclient: _finish_hunting 0 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 1 monclient: found mon.noname-c 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.924+0000 7fce77fff640 1 -- 172.21.15.110:0/3210708782 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce700094d0 con 0x7fce80142eb0 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.927+0000 7fce77fff640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce80142eb0 0x7fce800aac30 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7fce6800c7f0 tx=0x7fce6800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.927+0000 7fce76ffd640 10 client.?.objecter ms_handle_connect 0x7fce80142eb0 2024-01-27T04:20:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.927+0000 7fce76ffd640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 1 -- 172.21.15.110:0/3210708782 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68004220 con 0x7fce80142eb0 2024-01-27T04:20:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 10 monclient: dump: 2024-01-27T04:20:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 10 monclient: _finish_auth 0 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 10 monclient: _check_auth_tickets 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 1 -- 172.21.15.110:0/3210708782 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce68004380 con 0x7fce80142eb0 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:18.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 1 -- 172.21.15.110:0/3210708782 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68007860 con 0x7fce80142eb0 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce76ffd640 10 monclient: dump: 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:18.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce7d59a640 4 set_mon_vals no callback set 2024-01-27T04:20:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.930+0000 7fce84f2c640 5 monclient: authenticate success, global_id 18860 2024-01-27T04:20:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.931+0000 7fce84f2c640 10 monclient: _renew_subs 2024-01-27T04:20:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.931+0000 7fce84f2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.931+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce800ac1b0 con 0x7fce80142eb0 2024-01-27T04:20:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.931+0000 7fce84f2c640 10 client.18860.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.931+0000 7fce84f2c640 10 monclient: _renew_subs 2024-01-27T04:20:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.931+0000 7fce84f2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.931+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce800ad900 con 0x7fce80142eb0 2024-01-27T04:20:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.931+0000 7fce84f2c640 1 librados: init done 2024-01-27T04:20:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.931+0000 7fce84f2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.932+0000 7fce84f2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.932+0000 7fce84f2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.932+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce80074d10 con 0x7fce80142eb0 2024-01-27T04:20:18.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.951+0000 7fce76ffd640 1 -- 172.21.15.110:0/3210708782 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fce68007bd0 con 0x7fce80142eb0 2024-01-27T04:20:18.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.951+0000 7fce76ffd640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fce6006ca20 0x7fce6006eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:18.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.951+0000 7fce7e59c640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fce6006ca20 0x7fce6006eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:18.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.951+0000 7fce7e59c640 10 monclient: get_auth_request con 0x7fce6006ca20 auth_method 0 2024-01-27T04:20:18.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.952+0000 7fce76ffd640 10 client.18860.objecter ms_handle_connect 0x7fce6006ca20 2024-01-27T04:20:18.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:18.952+0000 7fce7e59c640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fce6006ca20 0x7fce6006eee0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fce70036000 tx=0x7fce7003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:19.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.188+0000 7fce76ffd640 1 -- 172.21.15.110:0/3210708782 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7fce68058660 con 0x7fce80142eb0 2024-01-27T04:20:19.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.188+0000 7fce76ffd640 10 client.18860.objecter ms_dispatch 0x7fce80000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:19.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.188+0000 7fce76ffd640 3 client.18860.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:19.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.188+0000 7fce76ffd640 3 client.18860.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:19.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.188+0000 7fce76ffd640 20 client.18860.objecter dump_active .. 0 homeless 2024-01-27T04:20:19.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.188+0000 7fce76ffd640 1 -- 172.21.15.110:0/3210708782 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fce68054900 con 0x7fce80142eb0 2024-01-27T04:20:19.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.188+0000 7fce76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:19.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.188+0000 7fce76ffd640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:19.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.365+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce8004ea50 con 0x7fce6006ca20 2024-01-27T04:20:19.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.370+0000 7fce76ffd640 1 -- 172.21.15.110:0/3210708782 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151868 (secure 0 0 0) 0x7fce8004ea50 con 0x7fce6006ca20 2024-01-27T04:20:19.373 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T04:20:19.373 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-27T04:20:19.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 10 librados: watch_flush enter 2024-01-27T04:20:19.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 10 librados: watch_flush exit 2024-01-27T04:20:19.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 20 client.18860.objecter shutdown clearing up homeless session... 2024-01-27T04:20:19.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 10 client.18860.objecter successfully canceled tick 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fce6006ca20 msgr2=0x7fce6006eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fce6006ca20 0x7fce6006eee0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fce70036000 tx=0x7fce7003d040 comp rx=0 tx=0).stop 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 10 monclient: shutdown 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce80142eb0 msgr2=0x7fce800aac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce80142eb0 0x7fce800aac30 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7fce6800c7f0 tx=0x7fce6800ccb0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 shutdown_connections 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fce6006ca20 0x7fce6006eee0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce80142eb0 0x7fce800aac30 unknown :-1 s=CLOSED pgs=3407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce800ab810 0x7fce800abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 --2- 172.21.15.110:0/3210708782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce800b4680 0x7fce8013ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 >> 172.21.15.110:0/3210708782 conn(0x7fce80066490 msgr2=0x7fce800b57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 shutdown_connections 2024-01-27T04:20:19.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 -- 172.21.15.110:0/3210708782 wait complete. 2024-01-27T04:20:19.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.372+0000 7fce84f2c640 1 librados: shutdown 2024-01-27T04:20:19.384 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":2451,"stamp":"2024-01-27T04:20:17.553087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":578036,"kb_used_data":39808,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754396684,"statfs":{"total":773094113280,"available":772502204416,"internally_reserved":0,"allocated":40763392,"data_stored":19076953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":151,"apply_latency_ms":151,"commit_latency_ns":151000000,"apply_latency_ns":151000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006263"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2557,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.713917+0000","last_change":"2024-01-27T03:45:12.104231+0000","last_active":"2024-01-27T04:20:16.713917+0000","last_peered":"2024-01-27T04:20:16.713917+0000","last_clean":"2024-01-27T04:20:16.713917+0000","last_became_active":"2024-01-27T03:45:12.095646+0000","last_became_peered":"2024-01-27T03:45:12.095646+0000","last_unstale":"2024-01-27T04:20:16.713917+0000","last_undegraded":"2024-01-27T04:20:16.713917+0000","last_fullsized":"2024-01-27T04:20:16.713917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:14:48.806131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2561,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.335104+0000","last_change":"2024-01-27T03:45:50.347781+0000","last_active":"2024-01-27T04:20:12.335104+0000","last_peered":"2024-01-27T04:20:12.335104+0000","last_clean":"2024-01-27T04:20:12.335104+0000","last_became_active":"2024-01-27T03:45:50.339311+0000","last_became_peered":"2024-01-27T03:45:50.339311+0000","last_unstale":"2024-01-27T04:20:12.335104+0000","last_undegraded":"2024-01-27T04:20:12.335104+0000","last_fullsized":"2024-01-27T04:20:12.335104+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:32.762753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2560,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.761038+0000","last_change":"2024-01-27T03:45:01.242765+0000","last_active":"2024-01-27T04:20:15.761038+0000","last_peered":"2024-01-27T04:20:15.761038+0000","last_clean":"2024-01-27T04:20:15.761038+0000","last_became_active":"2024-01-27T03:45:01.173317+0000","last_became_peered":"2024-01-27T03:45:01.173317+0000","last_unstale":"2024-01-27T04:20:15.761038+0000","last_undegraded":"2024-01-27T04:20:15.761038+0000","last_fullsized":"2024-01-27T04:20:15.761038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:45:07.062762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2547,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.781859+0000","last_change":"2024-01-27T03:45:08.326818+0000","last_active":"2024-01-27T04:20:13.781859+0000","last_peered":"2024-01-27T04:20:13.781859+0000","last_clean":"2024-01-27T04:20:13.781859+0000","last_became_active":"2024-01-27T03:45:08.187722+0000","last_became_peered":"2024-01-27T03:45:08.187722+0000","last_unstale":"2024-01-27T04:20:13.781859+0000","last_undegraded":"2024-01-27T04:20:13.781859+0000","last_fullsized":"2024-01-27T04:20:13.781859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:07:16.791137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2548,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.875693+0000","last_change":"2024-01-27T03:45:03.762871+0000","last_active":"2024-01-27T04:20:13.875693+0000","last_peered":"2024-01-27T04:20:13.875693+0000","last_clean":"2024-01-27T04:20:13.875693+0000","last_became_active":"2024-01-27T03:45:03.757221+0000","last_became_peered":"2024-01-27T03:45:03.757221+0000","last_unstale":"2024-01-27T04:20:13.875693+0000","last_undegraded":"2024-01-27T04:20:13.875693+0000","last_fullsized":"2024-01-27T04:20:13.875693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:52.156074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2572,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.736771+0000","last_change":"2024-01-27T03:45:01.929664+0000","last_active":"2024-01-27T04:20:14.736771+0000","last_peered":"2024-01-27T04:20:14.736771+0000","last_clean":"2024-01-27T04:20:14.736771+0000","last_became_active":"2024-01-27T03:45:01.916892+0000","last_became_peered":"2024-01-27T03:45:01.916892+0000","last_unstale":"2024-01-27T04:20:14.736771+0000","last_undegraded":"2024-01-27T04:20:14.736771+0000","last_fullsized":"2024-01-27T04:20:14.736771+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:02:49.278494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2561,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.352601+0000","last_change":"2024-01-27T03:45:50.416152+0000","last_active":"2024-01-27T04:20:12.352601+0000","last_peered":"2024-01-27T04:20:12.352601+0000","last_clean":"2024-01-27T04:20:12.352601+0000","last_became_active":"2024-01-27T03:45:50.401687+0000","last_became_peered":"2024-01-27T03:45:50.401687+0000","last_unstale":"2024-01-27T04:20:12.352601+0000","last_undegraded":"2024-01-27T04:20:12.352601+0000","last_fullsized":"2024-01-27T04:20:12.352601+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:34:59.292181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2563,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.357036+0000","last_change":"2024-01-27T03:45:16.267960+0000","last_active":"2024-01-27T04:20:12.357036+0000","last_peered":"2024-01-27T04:20:12.357036+0000","last_clean":"2024-01-27T04:20:12.357036+0000","last_became_active":"2024-01-27T03:45:16.039935+0000","last_became_peered":"2024-01-27T03:45:16.039935+0000","last_unstale":"2024-01-27T04:20:12.357036+0000","last_undegraded":"2024-01-27T04:20:12.357036+0000","last_fullsized":"2024-01-27T04:20:12.357036+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:02:05.330053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"48'7","reported_seq":2588,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.717267+0000","last_change":"2024-01-27T03:45:00.305112+0000","last_active":"2024-01-27T04:20:14.717267+0000","last_peered":"2024-01-27T04:20:14.717267+0000","last_clean":"2024-01-27T04:20:14.717267+0000","last_became_active":"2024-01-27T03:45:00.291987+0000","last_became_peered":"2024-01-27T03:45:00.291987+0000","last_unstale":"2024-01-27T04:20:14.717267+0000","last_undegraded":"2024-01-27T04:20:14.717267+0000","last_fullsized":"2024-01-27T04:20:14.717267+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:37.876016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2550,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.745980+0000","last_change":"2024-01-27T03:44:59.103125+0000","last_active":"2024-01-27T04:20:13.745980+0000","last_peered":"2024-01-27T04:20:13.745980+0000","last_clean":"2024-01-27T04:20:13.745980+0000","last_became_active":"2024-01-27T03:44:59.080371+0000","last_became_peered":"2024-01-27T03:44:59.080371+0000","last_unstale":"2024-01-27T04:20:13.745980+0000","last_undegraded":"2024-01-27T04:20:13.745980+0000","last_fullsized":"2024-01-27T04:20:13.745980+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:17.291036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2538,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.585376+0000","last_change":"2024-01-27T03:45:05.767658+0000","last_active":"2024-01-27T04:20:15.585376+0000","last_peered":"2024-01-27T04:20:15.585376+0000","last_clean":"2024-01-27T04:20:15.585376+0000","last_became_active":"2024-01-27T03:45:05.758355+0000","last_became_peered":"2024-01-27T03:45:05.758355+0000","last_unstale":"2024-01-27T04:20:15.585376+0000","last_undegraded":"2024-01-27T04:20:15.585376+0000","last_fullsized":"2024-01-27T04:20:15.585376+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:29:16.853701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2555,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.721456+0000","last_change":"2024-01-27T03:45:11.894875+0000","last_active":"2024-01-27T04:20:16.721456+0000","last_peered":"2024-01-27T04:20:16.721456+0000","last_clean":"2024-01-27T04:20:16.721456+0000","last_became_active":"2024-01-27T03:45:11.884947+0000","last_became_peered":"2024-01-27T03:45:11.884947+0000","last_unstale":"2024-01-27T04:20:16.721456+0000","last_undegraded":"2024-01-27T04:20:16.721456+0000","last_fullsized":"2024-01-27T04:20:16.721456+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:23:21.445636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":2577,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.279707+0000","last_change":"2024-01-27T03:43:25.035922+0000","last_active":"2024-01-27T04:20:12.279707+0000","last_peered":"2024-01-27T04:20:12.279707+0000","last_clean":"2024-01-27T04:20:12.279707+0000","last_became_active":"2024-01-27T03:43:24.404196+0000","last_became_peered":"2024-01-27T03:43:24.404196+0000","last_unstale":"2024-01-27T04:20:12.279707+0000","last_undegraded":"2024-01-27T04:20:12.279707+0000","last_fullsized":"2024-01-27T04:20:12.279707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_clean_scrub_stamp":"2024-01-27T03:43:13.902947+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:56:03.692684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2581,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.460782+0000","last_change":"2024-01-27T03:43:30.313647+0000","last_active":"2024-01-27T04:20:16.460782+0000","last_peered":"2024-01-27T04:20:16.460782+0000","last_clean":"2024-01-27T04:20:16.460782+0000","last_became_active":"2024-01-27T03:43:30.303196+0000","last_became_peered":"2024-01-27T03:43:30.303196+0000","last_unstale":"2024-01-27T04:20:16.460782+0000","last_undegraded":"2024-01-27T04:20:16.460782+0000","last_fullsized":"2024-01-27T04:20:16.460782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:45:18.950474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2549,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.750175+0000","last_change":"2024-01-27T03:45:06.207200+0000","last_active":"2024-01-27T04:20:13.750175+0000","last_peered":"2024-01-27T04:20:13.750175+0000","last_clean":"2024-01-27T04:20:13.750175+0000","last_became_active":"2024-01-27T03:45:06.197607+0000","last_became_peered":"2024-01-27T03:45:06.197607+0000","last_unstale":"2024-01-27T04:20:13.750175+0000","last_undegraded":"2024-01-27T04:20:13.750175+0000","last_fullsized":"2024-01-27T04:20:13.750175+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:30:59.795065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2583,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.446781+0000","last_change":"2024-01-27T03:43:30.123281+0000","last_active":"2024-01-27T04:20:16.446781+0000","last_peered":"2024-01-27T04:20:16.446781+0000","last_clean":"2024-01-27T04:20:16.446781+0000","last_became_active":"2024-01-27T03:43:30.033663+0000","last_became_peered":"2024-01-27T03:43:30.033663+0000","last_unstale":"2024-01-27T04:20:16.446781+0000","last_undegraded":"2024-01-27T04:20:16.446781+0000","last_fullsized":"2024-01-27T04:20:16.446781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:39:03.978730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2548,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.773182+0000","last_change":"2024-01-27T03:45:07.001823+0000","last_active":"2024-01-27T04:20:13.773182+0000","last_peered":"2024-01-27T04:20:13.773182+0000","last_clean":"2024-01-27T04:20:13.773182+0000","last_became_active":"2024-01-27T03:45:06.986316+0000","last_became_peered":"2024-01-27T03:45:06.986316+0000","last_unstale":"2024-01-27T04:20:13.773182+0000","last_undegraded":"2024-01-27T04:20:13.773182+0000","last_fullsized":"2024-01-27T04:20:13.773182+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:42.117379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2584,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.718217+0000","last_change":"2024-01-27T03:43:30.100170+0000","last_active":"2024-01-27T04:20:14.718217+0000","last_peered":"2024-01-27T04:20:14.718217+0000","last_clean":"2024-01-27T04:20:14.718217+0000","last_became_active":"2024-01-27T03:43:30.072010+0000","last_became_peered":"2024-01-27T03:43:30.072010+0000","last_unstale":"2024-01-27T04:20:14.718217+0000","last_undegraded":"2024-01-27T04:20:14.718217+0000","last_fullsized":"2024-01-27T04:20:14.718217+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:17:46.003541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2553,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.294627+0000","last_change":"2024-01-27T03:45:28.248666+0000","last_active":"2024-01-27T04:20:12.294627+0000","last_peered":"2024-01-27T04:20:12.294627+0000","last_clean":"2024-01-27T04:20:12.294627+0000","last_became_active":"2024-01-27T03:45:28.239949+0000","last_became_peered":"2024-01-27T03:45:28.239949+0000","last_unstale":"2024-01-27T04:20:12.294627+0000","last_undegraded":"2024-01-27T04:20:12.294627+0000","last_fullsized":"2024-01-27T04:20:12.294627+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:11:52.211119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.458866+0000","last_change":"2024-01-27T03:43:30.335032+0000","last_active":"2024-01-27T04:20:16.458866+0000","last_peered":"2024-01-27T04:20:16.458866+0000","last_clean":"2024-01-27T04:20:16.458866+0000","last_became_active":"2024-01-27T03:43:30.329967+0000","last_became_peered":"2024-01-27T03:43:30.329967+0000","last_unstale":"2024-01-27T04:20:16.458866+0000","last_undegraded":"2024-01-27T04:20:16.458866+0000","last_fullsized":"2024-01-27T04:20:16.458866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:58:30.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2549,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.741986+0000","last_change":"2024-01-27T03:45:03.469717+0000","last_active":"2024-01-27T04:20:13.741986+0000","last_peered":"2024-01-27T04:20:13.741986+0000","last_clean":"2024-01-27T04:20:13.741986+0000","last_became_active":"2024-01-27T03:45:03.435938+0000","last_became_peered":"2024-01-27T03:45:03.435938+0000","last_unstale":"2024-01-27T04:20:13.741986+0000","last_undegraded":"2024-01-27T04:20:13.741986+0000","last_fullsized":"2024-01-27T04:20:13.741986+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:31:47.895364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2559,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.872098+0000","last_change":"2024-01-27T03:43:30.340472+0000","last_active":"2024-01-27T04:20:13.872098+0000","last_peered":"2024-01-27T04:20:13.872098+0000","last_clean":"2024-01-27T04:20:13.872098+0000","last_became_active":"2024-01-27T03:43:30.329560+0000","last_became_peered":"2024-01-27T03:43:30.329560+0000","last_unstale":"2024-01-27T04:20:13.872098+0000","last_undegraded":"2024-01-27T04:20:13.872098+0000","last_fullsized":"2024-01-27T04:20:13.872098+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:10:58.217001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2560,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.350888+0000","last_change":"2024-01-27T03:45:11.682176+0000","last_active":"2024-01-27T04:20:12.350888+0000","last_peered":"2024-01-27T04:20:12.350888+0000","last_clean":"2024-01-27T04:20:12.350888+0000","last_became_active":"2024-01-27T03:45:11.344546+0000","last_became_peered":"2024-01-27T03:45:11.344546+0000","last_unstale":"2024-01-27T04:20:12.350888+0000","last_undegraded":"2024-01-27T04:20:12.350888+0000","last_fullsized":"2024-01-27T04:20:12.350888+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:35:01.350866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2581,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.473814+0000","last_change":"2024-01-27T03:43:30.077908+0000","last_active":"2024-01-27T04:20:16.473814+0000","last_peered":"2024-01-27T04:20:16.473814+0000","last_clean":"2024-01-27T04:20:16.473814+0000","last_became_active":"2024-01-27T03:43:30.057792+0000","last_became_peered":"2024-01-27T03:43:30.057792+0000","last_unstale":"2024-01-27T04:20:16.473814+0000","last_undegraded":"2024-01-27T04:20:16.473814+0000","last_fullsized":"2024-01-27T04:20:16.473814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:25:05.446156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2562,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.345492+0000","last_change":"2024-01-27T03:45:12.756405+0000","last_active":"2024-01-27T04:20:12.345492+0000","last_peered":"2024-01-27T04:20:12.345492+0000","last_clean":"2024-01-27T04:20:12.345492+0000","last_became_active":"2024-01-27T03:45:12.751200+0000","last_became_peered":"2024-01-27T03:45:12.751200+0000","last_unstale":"2024-01-27T04:20:12.345492+0000","last_undegraded":"2024-01-27T04:20:12.345492+0000","last_fullsized":"2024-01-27T04:20:12.345492+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:07:39.859584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2571,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.756811+0000","last_change":"2024-01-27T03:43:30.372081+0000","last_active":"2024-01-27T04:20:15.756811+0000","last_peered":"2024-01-27T04:20:15.756811+0000","last_clean":"2024-01-27T04:20:15.756811+0000","last_became_active":"2024-01-27T03:43:30.335129+0000","last_became_peered":"2024-01-27T03:43:30.335129+0000","last_unstale":"2024-01-27T04:20:15.756811+0000","last_undegraded":"2024-01-27T04:20:15.756811+0000","last_fullsized":"2024-01-27T04:20:15.756811+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:38:01.568710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2570,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.472197+0000","last_change":"2024-01-27T03:45:11.743478+0000","last_active":"2024-01-27T04:20:16.472197+0000","last_peered":"2024-01-27T04:20:16.472197+0000","last_clean":"2024-01-27T04:20:16.472197+0000","last_became_active":"2024-01-27T03:45:11.734547+0000","last_became_peered":"2024-01-27T03:45:11.734547+0000","last_unstale":"2024-01-27T04:20:16.472197+0000","last_undegraded":"2024-01-27T04:20:16.472197+0000","last_fullsized":"2024-01-27T04:20:16.472197+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T03:53:58.163231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2579,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.468607+0000","last_change":"2024-01-27T03:43:30.350497+0000","last_active":"2024-01-27T04:20:16.468607+0000","last_peered":"2024-01-27T04:20:16.468607+0000","last_clean":"2024-01-27T04:20:16.468607+0000","last_became_active":"2024-01-27T03:43:30.345375+0000","last_became_peered":"2024-01-27T03:43:30.345375+0000","last_unstale":"2024-01-27T04:20:16.468607+0000","last_undegraded":"2024-01-27T04:20:16.468607+0000","last_fullsized":"2024-01-27T04:20:16.468607+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:25:58.675410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2570,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:14.735085+0000","last_change":"2024-01-27T03:45:08.095965+0000","last_active":"2024-01-27T04:20:14.735085+0000","last_peered":"2024-01-27T04:20:14.735085+0000","last_clean":"2024-01-27T04:20:14.735085+0000","last_became_active":"2024-01-27T03:45:08.074261+0000","last_became_peered":"2024-01-27T03:45:08.074261+0000","last_unstale":"2024-01-27T04:20:14.735085+0000","last_undegraded":"2024-01-27T04:20:14.735085+0000","last_fullsized":"2024-01-27T04:20:14.735085+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T15:20:03.504139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2560,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.733166+0000","last_change":"2024-01-27T03:45:29.625526+0000","last_active":"2024-01-27T04:20:16.733166+0000","last_peered":"2024-01-27T04:20:16.733166+0000","last_clean":"2024-01-27T04:20:16.733166+0000","last_became_active":"2024-01-27T03:45:29.620303+0000","last_became_peered":"2024-01-27T03:45:29.620303+0000","last_unstale":"2024-01-27T04:20:16.733166+0000","last_undegraded":"2024-01-27T04:20:16.733166+0000","last_fullsized":"2024-01-27T04:20:16.733166+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:50:09.679551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2554,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.737195+0000","last_change":"2024-01-27T03:45:11.908907+0000","last_active":"2024-01-27T04:20:16.737195+0000","last_peered":"2024-01-27T04:20:16.737195+0000","last_clean":"2024-01-27T04:20:16.737195+0000","last_became_active":"2024-01-27T03:45:11.902875+0000","last_became_peered":"2024-01-27T03:45:11.902875+0000","last_unstale":"2024-01-27T04:20:16.737195+0000","last_undegraded":"2024-01-27T04:20:16.737195+0000","last_fullsized":"2024-01-27T04:20:16.737195+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T14:33:16.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2547,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.831239+0000","last_change":"2024-01-27T03:45:41.083249+0000","last_active":"2024-01-27T04:20:13.831239+0000","last_peered":"2024-01-27T04:20:13.831239+0000","last_clean":"2024-01-27T04:20:13.831239+0000","last_became_active":"2024-01-27T03:45:41.077954+0000","last_became_peered":"2024-01-27T03:45:41.077954+0000","last_unstale":"2024-01-27T04:20:13.831239+0000","last_undegraded":"2024-01-27T04:20:13.831239+0000","last_fullsized":"2024-01-27T04:20:13.831239+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:53:19.777797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2555,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.730962+0000","last_change":"2024-01-27T03:45:13.508628+0000","last_active":"2024-01-27T04:20:16.730962+0000","last_peered":"2024-01-27T04:20:16.730962+0000","last_clean":"2024-01-27T04:20:16.730962+0000","last_became_active":"2024-01-27T03:45:13.503336+0000","last_became_peered":"2024-01-27T03:45:13.503336+0000","last_unstale":"2024-01-27T04:20:16.730962+0000","last_undegraded":"2024-01-27T04:20:16.730962+0000","last_fullsized":"2024-01-27T04:20:16.730962+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:08:42.711117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2551,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.291640+0000","last_change":"2024-01-27T03:45:06.421706+0000","last_active":"2024-01-27T04:20:12.291640+0000","last_peered":"2024-01-27T04:20:12.291640+0000","last_clean":"2024-01-27T04:20:12.291640+0000","last_became_active":"2024-01-27T03:45:06.412514+0000","last_became_peered":"2024-01-27T03:45:06.412514+0000","last_unstale":"2024-01-27T04:20:12.291640+0000","last_undegraded":"2024-01-27T04:20:12.291640+0000","last_fullsized":"2024-01-27T04:20:12.291640+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:37:23.739645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2562,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.337984+0000","last_change":"2024-01-27T03:45:05.911725+0000","last_active":"2024-01-27T04:20:12.337984+0000","last_peered":"2024-01-27T04:20:12.337984+0000","last_clean":"2024-01-27T04:20:12.337984+0000","last_became_active":"2024-01-27T03:45:05.905013+0000","last_became_peered":"2024-01-27T03:45:05.905013+0000","last_unstale":"2024-01-27T04:20:12.337984+0000","last_undegraded":"2024-01-27T04:20:12.337984+0000","last_fullsized":"2024-01-27T04:20:12.337984+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:23:34.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2552,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.277864+0000","last_change":"2024-01-27T03:45:09.267062+0000","last_active":"2024-01-27T04:20:12.277864+0000","last_peered":"2024-01-27T04:20:12.277864+0000","last_clean":"2024-01-27T04:20:12.277864+0000","last_became_active":"2024-01-27T03:45:08.513799+0000","last_became_peered":"2024-01-27T03:45:08.513799+0000","last_unstale":"2024-01-27T04:20:12.277864+0000","last_undegraded":"2024-01-27T04:20:12.277864+0000","last_fullsized":"2024-01-27T04:20:12.277864+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:27:45.278278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2571,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.448883+0000","last_change":"2024-01-27T03:45:12.810545+0000","last_active":"2024-01-27T04:20:16.448883+0000","last_peered":"2024-01-27T04:20:16.448883+0000","last_clean":"2024-01-27T04:20:16.448883+0000","last_became_active":"2024-01-27T03:45:12.797603+0000","last_became_peered":"2024-01-27T03:45:12.797603+0000","last_unstale":"2024-01-27T04:20:16.448883+0000","last_undegraded":"2024-01-27T04:20:16.448883+0000","last_fullsized":"2024-01-27T04:20:16.448883+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:06:00.673252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2532,"reported_epoch":1332,"state":"active+clean","last_fresh":"2024-01-27T04:20:15.622160+0000","last_change":"2024-01-27T03:45:08.152790+0000","last_active":"2024-01-27T04:20:15.622160+0000","last_peered":"2024-01-27T04:20:15.622160+0000","last_clean":"2024-01-27T04:20:15.622160+0000","last_became_active":"2024-01-27T03:45:08.146505+0000","last_became_peered":"2024-01-27T03:45:08.146505+0000","last_unstale":"2024-01-27T04:20:15.622160+0000","last_undegraded":"2024-01-27T04:20:15.622160+0000","last_fullsized":"2024-01-27T04:20:15.622160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:59:50.587416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2559,"reported_epoch":1330,"state":"active+clean","last_fresh":"2024-01-27T04:20:12.349162+0000","last_change":"2024-01-27T03:45:10.396512+0000","last_active":"2024-01-27T04:20:12.349162+0000","last_peered":"2024-01-27T04:20:12.349162+0000","last_clean":"2024-01-27T04:20:12.349162+0000","last_became_active":"2024-01-27T03:45:10.387717+0000","last_became_peered":"2024-01-27T03:45:10.387717+0000","last_unstale":"2024-01-27T04:20:12.349162+0000","last_undegraded":"2024-01-27T04:20:12.349162+0000","last_fullsized":"2024-01-27T04:20:12.349162+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:12:16.534326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2548,"reported_epoch":1331,"state":"active+clean","last_fresh":"2024-01-27T04:20:13.785854+0000","last_change":"2024-01-27T03:45:41.124551+0000","last_active":"2024-01-27T04:20:13.785854+0000","last_peered":"2024-01-27T04:20:13.785854+0000","last_clean":"2024-01-27T04:20:13.785854+0000","last_became_active":"2024-01-27T03:45:41.119306+0000","last_became_peered":"2024-01-27T03:45:41.119306+0000","last_unstale":"2024-01-27T04:20:13.785854+0000","last_undegraded":"2024-01-27T04:20:13.785854+0000","last_fullsized":"2024-01-27T04:20:13.785854+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:52:56.858068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2569,"reported_epoch":1333,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.470531+0000","last_change":"2024-01-27T03:45:15.265604+0000","last_active":"2024-01-27T04:20:16.470531+0000","last_peered":"2024-01-27T04:20:16.470531+0000","last_clean":"2024-01-27T04:20:16.470531+0000","last_became_active":"2024-01-27T03:45:15.259193+0000","last_became_peered":"2024-01-27T03:45:15.259193+0000","last_unstale":"2024-01-27T04:20:16.470531+0000","last_undegraded":"2024-01-27T04:20:16.470531+0000","last_fullsized":"2024-01-27T04:20:16.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:33:48.948534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313662,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68372,"kb_used_data":5248,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303468,"statfs":{"total":96636764160,"available":96566751232,"internally_reserved":0,"allocated":5373952,"data_stored":2671367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":11.641999999999999,"5min":22.148,"15min":18.57},"min":{"1min":4.3710000000000004,"5min":4.3710000000000004,"15min":4.3710000000000004},"max":{"1min":35.859000000000002,"5min":232.63900000000001,"15min":232.63900000000001},"last":32.728999999999999},{"interface":"front","average":{"1min":15.961,"5min":23.016999999999999,"15min":19.32},"min":{"1min":4.8449999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"max":{"1min":48.558999999999997,"5min":265.815,"15min":265.815},"last":29.667999999999999}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.598000000000001,"5min":27.135000000000002,"15min":23.359999999999999},"min":{"1min":6.415,"5min":6.415,"15min":5.4749999999999996},"max":{"1min":55.116,"5min":152.09700000000001,"15min":421.90300000000002},"last":25.469999999999999},{"interface":"front","average":{"1min":12.83,"5min":26.446999999999999,"15min":21.591000000000001},"min":{"1min":6.3529999999999998,"5min":5.968,"15min":5.7039999999999997},"max":{"1min":40.463999999999999,"5min":165.53999999999999,"15min":319.13200000000001},"last":22.477}]},{"osd":3,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":16.913,"5min":26.349,"15min":24.771000000000001},"min":{"1min":7.9109999999999996,"5min":7.2750000000000004,"15min":6.0670000000000002},"max":{"1min":54.639000000000003,"5min":176.215,"15min":434.72800000000001},"last":37.106000000000002},{"interface":"front","average":{"1min":23.969000000000001,"5min":34.192999999999998,"15min":31.582999999999998},"min":{"1min":10.372999999999999,"5min":10.372999999999999,"15min":8.3019999999999996},"max":{"1min":95.001000000000005,"5min":216.80099999999999,"15min":447.06700000000001},"last":58.091999999999999}]},{"osd":4,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":19.324000000000002,"5min":24.849,"15min":23.100999999999999},"min":{"1min":8.2040000000000006,"5min":7.4240000000000004,"15min":7.0979999999999999},"max":{"1min":59.201000000000001,"5min":152.88,"15min":329.11500000000001},"last":38.07},{"interface":"front","average":{"1min":18.565999999999999,"5min":24.952999999999999,"15min":21.696999999999999},"min":{"1min":8.4870000000000001,"5min":7.1239999999999997,"15min":5.7839999999999998},"max":{"1min":61.326999999999998,"5min":233.845,"15min":233.845},"last":30.361999999999998}]},{"osd":5,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":25.951000000000001,"5min":30.747,"15min":26.004000000000001},"min":{"1min":8.7680000000000007,"5min":8.7680000000000007,"15min":7.165},"max":{"1min":101.246,"5min":252.73500000000001,"15min":324.52300000000002},"last":53.686},{"interface":"front","average":{"1min":27.847999999999999,"5min":38.902999999999999,"15min":30.933},"min":{"1min":13.746,"5min":11.076000000000001,"15min":9.0210000000000008},"max":{"1min":82.462999999999994,"5min":382,"15min":382},"last":35.701000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":29.972999999999999,"15min":28.385000000000002},"min":{"1min":7.335,"5min":7.335,"15min":7.335},"max":{"1min":33.107999999999997,"5min":165.976,"15min":400.93900000000002},"last":31.201000000000001},{"interface":"front","average":{"1min":16.268000000000001,"5min":28.919,"15min":27.251000000000001},"min":{"1min":9.4049999999999994,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":26.937999999999999,"5min":217.77699999999999,"15min":684.36699999999996},"last":34.392000000000003}]},{"osd":7,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":20.309999999999999,"5min":33.465000000000003,"15min":31.215},"min":{"1min":9.0519999999999996,"5min":8.1389999999999993,"15min":8.1389999999999993},"max":{"1min":50.585000000000001,"5min":242.08199999999999,"15min":443.21499999999997},"last":39.984000000000002},{"interface":"front","average":{"1min":27.695,"5min":32.735999999999997,"15min":28.863},"min":{"1min":12.496,"5min":9.6590000000000007,"15min":8.3309999999999995},"max":{"1min":133.63200000000001,"5min":145.751,"15min":436.80399999999997},"last":33.523000000000003}]}]},{"osd":2,"up_from":21,"seq":90194313660,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76292,"kb_used_data":4856,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295548,"statfs":{"total":96636764160,"available":96558641152,"internally_reserved":0,"allocated":4972544,"data_stored":2273527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":8.7110000000000003,"5min":25.661999999999999,"15min":18.187999999999999},"min":{"1min":4.2939999999999996,"5min":4.2939999999999996,"15min":4.2939999999999996},"max":{"1min":23.433,"5min":445.62400000000002,"15min":445.62400000000002},"last":16.015999999999998},{"interface":"front","average":{"1min":12.178000000000001,"5min":34.600999999999999,"15min":22.532},"min":{"1min":7.1420000000000003,"5min":6.4370000000000003,"15min":5.3230000000000004},"max":{"1min":16.050000000000001,"5min":974.43299999999999,"15min":974.43299999999999},"last":21.056999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.199999999999999,"5min":37.404000000000003,"15min":25.739000000000001},"min":{"1min":7.0250000000000004,"5min":7.0250000000000004,"15min":6.6239999999999997},"max":{"1min":19.93,"5min":975.43499999999995,"15min":975.43499999999995},"last":8.5619999999999994},{"interface":"front","average":{"1min":12.119999999999999,"5min":28.672000000000001,"15min":22.556000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":5.5899999999999999},"max":{"1min":25.190000000000001,"5min":481.185,"15min":481.185},"last":12.433999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":17.236999999999998,"5min":36.584000000000003,"15min":26.488},"min":{"1min":7.665,"5min":7.2709999999999999,"15min":7.2709999999999999},"max":{"1min":59.902999999999999,"5min":980.83500000000004,"15min":980.83500000000004},"last":11.738},{"interface":"front","average":{"1min":18.442,"5min":41.698,"15min":29.129999999999999},"min":{"1min":11.558999999999999,"5min":10.58,"15min":9.5459999999999994},"max":{"1min":26.449000000000002,"5min":980.17499999999995,"15min":980.17499999999995},"last":15.047000000000001}]},{"osd":4,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":12.173999999999999,"5min":27.407,"15min":22.308},"min":{"1min":6.7809999999999997,"5min":6.2750000000000004,"15min":5.819},"max":{"1min":23.806000000000001,"5min":508.601,"15min":508.601},"last":8.2010000000000005},{"interface":"front","average":{"1min":14.736000000000001,"5min":34.642000000000003,"15min":27.102},"min":{"1min":9.7059999999999995,"5min":8.234,"15min":6.6580000000000004},"max":{"1min":27.920999999999999,"5min":492.29199999999997,"15min":492.29199999999997},"last":7.8460000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":18.317,"5min":48.280000000000001,"15min":33.753999999999998},"min":{"1min":8.9800000000000004,"5min":8.9800000000000004,"15min":8.9740000000000002},"max":{"1min":26.623999999999999,"5min":976.35900000000004,"15min":976.35900000000004},"last":17.640999999999998},{"interface":"front","average":{"1min":12.997,"5min":29.919,"15min":23.949000000000002},"min":{"1min":9.1140000000000008,"5min":6.9610000000000003,"15min":5.758},"max":{"1min":18.027999999999999,"5min":509.71699999999998,"15min":509.71699999999998},"last":10.263}]},{"osd":6,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":33.555999999999997,"15min":28.885000000000002},"min":{"1min":10.654999999999999,"5min":7.8869999999999996,"15min":6.3390000000000004},"max":{"1min":22.977,"5min":526.51400000000001,"15min":526.51400000000001},"last":14.087},{"interface":"front","average":{"1min":16.966000000000001,"5min":39.537999999999997,"15min":35.607999999999997},"min":{"1min":12.301,"5min":10.784000000000001,"15min":9.4350000000000005},"max":{"1min":23.710999999999999,"5min":533.20799999999997,"15min":533.20799999999997},"last":19.152000000000001}]},{"osd":7,"last update":"Sat Jan 27 04:19:26 2024","interfaces":[{"interface":"back","average":{"1min":25.797999999999998,"5min":49.715000000000003,"15min":36.329000000000001},"min":{"1min":14.427,"5min":11.739000000000001,"15min":9.5139999999999993},"max":{"1min":74.819000000000003,"5min":977.13800000000003,"15min":977.13800000000003},"last":20.050999999999998},{"interface":"front","average":{"1min":18.117000000000001,"5min":33.683,"15min":29.026},"min":{"1min":11.587999999999999,"5min":8.5850000000000009,"15min":8.5850000000000009},"max":{"1min":52.927999999999997,"5min":549.30100000000004,"15min":549.30100000000004},"last":16.943999999999999}]}]},{"osd":6,"up_from":20,"seq":85899346368,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72180,"kb_used_data":4896,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299660,"statfs":{"total":96636764160,"available":96562851840,"internally_reserved":0,"allocated":5013504,"data_stored":2295222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":9.9730000000000008,"5min":22.446000000000002,"15min":18.524999999999999},"min":{"1min":5.492,"5min":4.6929999999999996,"15min":4.2350000000000003},"max":{"1min":20.303000000000001,"5min":339.68000000000001,"15min":339.68000000000001},"last":9.9979999999999993},{"interface":"front","average":{"1min":14.538,"5min":32.518000000000001,"15min":26.594999999999999},"min":{"1min":9.9329999999999998,"5min":5.5049999999999999,"15min":4.9749999999999996},"max":{"1min":27.638999999999999,"5min":341.03800000000001,"15min":341.03800000000001},"last":21.902999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":12.906000000000001,"5min":24.239999999999998,"15min":18.077999999999999},"min":{"1min":6.1100000000000003,"5min":5.1970000000000001,"15min":4.6950000000000003},"max":{"1min":35.095999999999997,"5min":250.96000000000001,"15min":250.96000000000001},"last":6.9530000000000003},{"interface":"front","average":{"1min":12.737,"5min":29.638999999999999,"15min":23.449000000000002},"min":{"1min":6.181,"5min":5.0679999999999996,"15min":4.7670000000000003},"max":{"1min":27.792000000000002,"5min":350.74599999999998,"15min":350.74599999999998},"last":8.3989999999999991}]},{"osd":2,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":12.606999999999999,"5min":31.344999999999999,"15min":23.207000000000001},"min":{"1min":6.6669999999999998,"5min":6.6669999999999998,"15min":6.0839999999999996},"max":{"1min":25.248999999999999,"5min":349.19299999999998,"15min":349.19299999999998},"last":18.024000000000001},{"interface":"front","average":{"1min":14.715,"5min":35.548000000000002,"15min":29.663},"min":{"1min":9.3729999999999993,"5min":6.5499999999999998,"15min":6.4630000000000001},"max":{"1min":23.071999999999999,"5min":354.80799999999999,"15min":354.80799999999999},"last":24.498999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:43 2024","interfaces":[{"interface":"back","average":{"1min":16.457000000000001,"5min":32.703000000000003,"15min":24.315000000000001},"min":{"1min":6.9299999999999997,"5min":6.9279999999999999,"15min":6.7469999999999999},"max":{"1min":54.694000000000003,"5min":247.446,"15min":265.50299999999999},"last":9.25},{"interface":"front","average":{"1min":20.056999999999999,"5min":34.040999999999997,"15min":28.632999999999999},"min":{"1min":10.635999999999999,"5min":6.4930000000000003,"15min":6.4930000000000003},"max":{"1min":49.180999999999997,"5min":341.68400000000003,"15min":352.79399999999998},"last":26.282}]},{"osd":4,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":15.965,"5min":27.364999999999998,"15min":24.411000000000001},"min":{"1min":10.279999999999999,"5min":6.7590000000000003,"15min":6.5519999999999996},"max":{"1min":33.625999999999998,"5min":243.96100000000001,"15min":266.12400000000002},"last":20.373000000000001},{"interface":"front","average":{"1min":14.163,"5min":30.145,"15min":27.082000000000001},"min":{"1min":7.359,"5min":6.6859999999999999,"15min":6.0289999999999999},"max":{"1min":29.823,"5min":348.11500000000001,"15min":348.11500000000001},"last":18.652999999999999}]},{"osd":5,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":17.280999999999999,"5min":30.981999999999999,"15min":26.800000000000001},"min":{"1min":11.436999999999999,"5min":7.827,"15min":7.827},"max":{"1min":28.574000000000002,"5min":295.56,"15min":295.56},"last":28.443999999999999},{"interface":"front","average":{"1min":19.364000000000001,"5min":37.106000000000002,"15min":30.992000000000001},"min":{"1min":11.782,"5min":10.959,"15min":8.3569999999999993},"max":{"1min":47.966999999999999,"5min":349.81,"15min":354.55099999999999},"last":29.861999999999998}]},{"osd":7,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":26.315000000000001,"5min":38.106000000000002,"15min":25.777000000000001},"min":{"1min":14.762,"5min":9.7609999999999992,"15min":7.4199999999999999},"max":{"1min":93.060000000000002,"5min":346.697,"15min":346.697},"last":26.91},{"interface":"front","average":{"1min":18.989999999999998,"5min":37.228000000000002,"15min":28.795000000000002},"min":{"1min":11.340999999999999,"5min":7.7670000000000003,"15min":7.7670000000000003},"max":{"1min":49.277000000000001,"5min":352.11900000000003,"15min":369.92899999999997},"last":19.280000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346365,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72164,"kb_used_data":4888,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299676,"statfs":{"total":96636764160,"available":96562868224,"internally_reserved":0,"allocated":5005312,"data_stored":2288496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":12.260999999999999,"5min":21.777000000000001,"15min":17.795999999999999},"min":{"1min":4.5209999999999999,"5min":4.4020000000000001,"15min":3.782},"max":{"1min":30.248000000000001,"5min":271.60599999999999,"15min":271.60599999999999},"last":30.248000000000001},{"interface":"front","average":{"1min":11.218999999999999,"5min":19.111999999999998,"15min":15.164999999999999},"min":{"1min":4.8700000000000001,"5min":4.8700000000000001,"15min":4.7830000000000004},"max":{"1min":36.665999999999997,"5min":141.624,"15min":141.624},"last":22.373999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":15.82,"5min":20.286999999999999,"15min":20.946000000000002},"min":{"1min":7.3730000000000002,"5min":5.8159999999999998,"15min":5.8159999999999998},"max":{"1min":44.912999999999997,"5min":228.27600000000001,"15min":228.27600000000001},"last":32.338000000000001},{"interface":"front","average":{"1min":13.35,"5min":23.087,"15min":21.771999999999998},"min":{"1min":7.7060000000000004,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":41.997,"5min":276.93299999999999,"15min":276.93299999999999},"last":28.045000000000002}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":18.422999999999998,"5min":21.227,"15min":22.727},"min":{"1min":7.9669999999999996,"5min":7.8319999999999999,"15min":6.4219999999999997},"max":{"1min":62.054000000000002,"5min":138.13300000000001,"15min":252.38399999999999},"last":30.907},{"interface":"front","average":{"1min":16.931999999999999,"5min":26.645,"15min":22.036999999999999},"min":{"1min":9.7780000000000005,"5min":7.4930000000000003,"15min":5.8979999999999997},"max":{"1min":48.671999999999997,"5min":305.358,"15min":321.09399999999999},"last":23.082999999999998}]},{"osd":3,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":19.376000000000001,"5min":22.815999999999999,"15min":22.629999999999999},"min":{"1min":10.507999999999999,"5min":8.452,"15min":8.1780000000000008},"max":{"1min":61.652999999999999,"5min":89.129999999999995,"15min":195.47800000000001},"last":21.466999999999999},{"interface":"front","average":{"1min":14.627000000000001,"5min":17.146999999999998,"15min":16.039999999999999},"min":{"1min":6.4699999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":91.978999999999999,"5min":120.044,"15min":154.279},"last":7.0910000000000002}]},{"osd":5,"last update":"Sat Jan 27 04:19:38 2024","interfaces":[{"interface":"back","average":{"1min":16.870999999999999,"5min":26.902000000000001,"15min":21.398},"min":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.1779999999999999},"max":{"1min":57.954999999999998,"5min":243.983,"15min":243.983},"last":43.405999999999999},{"interface":"front","average":{"1min":21.181000000000001,"5min":27.506,"15min":24.664000000000001},"min":{"1min":11.605,"5min":9.7690000000000001,"15min":9.7200000000000006},"max":{"1min":61.289000000000001,"5min":188.453,"15min":191.036},"last":59.655000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":21.027000000000001,"5min":29.692,"15min":27.587},"min":{"1min":8.3409999999999993,"5min":8.3409999999999993,"15min":8.3409999999999993},"max":{"1min":46.170999999999999,"5min":259.351,"15min":259.351},"last":8.3409999999999993},{"interface":"front","average":{"1min":24.855,"5min":30.391999999999999,"15min":24.253},"min":{"1min":14.805,"5min":6.9089999999999998,"15min":6.4630000000000001},"max":{"1min":49.533999999999999,"5min":295.00900000000001,"15min":295.00900000000001},"last":25.052}]},{"osd":7,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":20.744,"5min":25.904,"15min":21.736999999999998},"min":{"1min":10.656000000000001,"5min":8.1850000000000005,"15min":6.4269999999999996},"max":{"1min":52.637999999999998,"5min":206.73400000000001,"15min":206.73400000000001},"last":9.5030000000000001},{"interface":"front","average":{"1min":26.753,"5min":32.119,"15min":27.643000000000001},"min":{"1min":15.661,"5min":10.401999999999999,"15min":9.7940000000000005},"max":{"1min":90.658000000000001,"5min":262.28300000000002,"15min":262.28300000000002},"last":29.603999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477181,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72164,"kb_used_data":4884,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299676,"statfs":{"total":96636764160,"available":96562868224,"internally_reserved":0,"allocated":5001216,"data_stored":2288491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:36 2024","interfaces":[{"interface":"back","average":{"1min":11.134,"5min":16.122,"15min":16.141999999999999},"min":{"1min":5.3609999999999998,"5min":5.3609999999999998,"15min":4.6310000000000002},"max":{"1min":33.012,"5min":93.953000000000003,"15min":238.642},"last":13.462},{"interface":"front","average":{"1min":14.760999999999999,"5min":20.975000000000001,"15min":20.623999999999999},"min":{"1min":8.0269999999999992,"5min":7.274,"15min":7.274},"max":{"1min":34.912999999999997,"5min":135.18299999999999,"15min":230.04900000000001},"last":14.769}]},{"osd":1,"last update":"Sat Jan 27 04:19:16 2024","interfaces":[{"interface":"back","average":{"1min":10.414,"5min":15.129,"15min":15.5},"min":{"1min":6.0369999999999999,"5min":5.0730000000000004,"15min":4.7359999999999998},"max":{"1min":27.911000000000001,"5min":103.377,"15min":188.899},"last":6.1779999999999999},{"interface":"front","average":{"1min":10.236000000000001,"5min":16.545000000000002,"15min":20.106000000000002},"min":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":30.829000000000001,"5min":102.009,"15min":218.97399999999999},"last":18.475999999999999}]},{"osd":2,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":16.824000000000002,"5min":22.969999999999999,"15min":21.879999999999999},"min":{"1min":7.8319999999999999,"5min":7.8319999999999999,"15min":7.8319999999999999},"max":{"1min":44.707000000000001,"5min":130.16499999999999,"15min":322.24599999999998},"last":16.512},{"interface":"front","average":{"1min":19.797999999999998,"5min":25.126000000000001,"15min":25.841999999999999},"min":{"1min":11,"5min":9.8699999999999992,"15min":9.8699999999999992},"max":{"1min":53.359000000000002,"5min":201.851,"15min":326.04700000000003},"last":20.082999999999998}]},{"osd":4,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":22.923999999999999,"5min":27.471,"15min":25.827000000000002},"min":{"1min":11.742000000000001,"5min":10.131,"15min":10.131},"max":{"1min":115.759,"5min":220.51300000000001,"15min":250.255},"last":19.363},{"interface":"front","average":{"1min":19.719999999999999,"5min":20.579999999999998,"15min":17.524999999999999},"min":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.5049999999999999},"max":{"1min":150.36199999999999,"5min":221.44999999999999,"15min":227.67400000000001},"last":7.9390000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:19:29 2024","interfaces":[{"interface":"back","average":{"1min":15.736000000000001,"5min":17.591000000000001,"15min":15.472},"min":{"1min":7.2169999999999996,"5min":5.5979999999999999,"15min":4.9359999999999999},"max":{"1min":49.698999999999998,"5min":212.291,"15min":212.291},"last":17.169},{"interface":"front","average":{"1min":16.992999999999999,"5min":19.856999999999999,"15min":19.529},"min":{"1min":8.782,"5min":6.742,"15min":6.3550000000000004},"max":{"1min":70.760000000000005,"5min":147.773,"15min":237.16300000000001},"last":15.65}]},{"osd":6,"last update":"Sat Jan 27 04:19:57 2024","interfaces":[{"interface":"back","average":{"1min":15.177,"5min":20.619,"15min":19.541},"min":{"1min":8.1270000000000007,"5min":7.4939999999999998,"15min":6.1760000000000002},"max":{"1min":27.283999999999999,"5min":145.55199999999999,"15min":261.56799999999998},"last":12.894},{"interface":"front","average":{"1min":16.452000000000002,"5min":23.617999999999999,"15min":23.219999999999999},"min":{"1min":11.131,"5min":7.8700000000000001,"15min":7.8700000000000001},"max":{"1min":34.841999999999999,"5min":135.69900000000001,"15min":276.86099999999999},"last":10.686}]},{"osd":7,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":27.474,"5min":29.882999999999999,"15min":26.277999999999999},"min":{"1min":12.359999999999999,"5min":8.0449999999999999,"15min":8.0449999999999999},"max":{"1min":188.46100000000001,"5min":188.46100000000001,"15min":382.57999999999998},"last":22.079000000000001},{"interface":"front","average":{"1min":24.861000000000001,"5min":28.331,"15min":22.138999999999999},"min":{"1min":9.2249999999999996,"5min":9.2249999999999996,"15min":8.9849999999999994},"max":{"1min":222.232,"5min":222.232,"15min":254.12},"last":10.050000000000001}]}]},{"osd":0,"up_from":16,"seq":68719477179,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72160,"kb_used_data":4884,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299680,"statfs":{"total":96636764160,"available":96562872320,"internally_reserved":0,"allocated":5001216,"data_stored":2288491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":11.346,"5min":21.847999999999999,"15min":19.535},"min":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"max":{"1min":28.620000000000001,"5min":197.15600000000001,"15min":286.24799999999999},"last":9.8729999999999993},{"interface":"front","average":{"1min":13.196999999999999,"5min":29.672999999999998,"15min":23.280999999999999},"min":{"1min":6.726,"5min":6.0350000000000001,"15min":4.7960000000000003},"max":{"1min":36.061,"5min":254.524,"15min":267.99299999999999},"last":8.3680000000000003}]},{"osd":2,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":12.795999999999999,"5min":24.466999999999999,"15min":21.399000000000001},"min":{"1min":6.4829999999999997,"5min":5.984,"15min":5.1449999999999996},"max":{"1min":19.736000000000001,"5min":229.99299999999999,"15min":330.28699999999998},"last":15.855},{"interface":"front","average":{"1min":16.109999999999999,"5min":28.257000000000001,"15min":22.437999999999999},"min":{"1min":10.855,"5min":7.6020000000000003,"15min":6.1669999999999998},"max":{"1min":30.111999999999998,"5min":218.97900000000001,"15min":289.72899999999998},"last":8.9809999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:27 2024","interfaces":[{"interface":"back","average":{"1min":12.762,"5min":23.349,"15min":19.062999999999999},"min":{"1min":6.3230000000000004,"5min":6.3230000000000004,"15min":6.1840000000000002},"max":{"1min":28.111999999999998,"5min":156.90100000000001,"15min":243.779},"last":7.4039999999999999},{"interface":"front","average":{"1min":19.774999999999999,"5min":31.922999999999998,"15min":25.346},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.0860000000000003},"max":{"1min":32.655999999999999,"5min":265.04300000000001,"15min":265.04300000000001},"last":14.35}]},{"osd":4,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":22.652000000000001,"5min":31.033000000000001,"15min":24.866},"min":{"1min":8.2270000000000003,"5min":7.4429999999999996,"15min":7.1870000000000003},"max":{"1min":90.774000000000001,"5min":238.68600000000001,"15min":238.68600000000001},"last":10.619999999999999},{"interface":"front","average":{"1min":26.318999999999999,"5min":30.364999999999998,"15min":27.841000000000001},"min":{"1min":10.281000000000001,"5min":9.7170000000000005,"15min":9.5559999999999992},"max":{"1min":89.200000000000003,"5min":174.03800000000001,"15min":242.22800000000001},"last":16.584}]},{"osd":5,"last update":"Sat Jan 27 04:19:27 2024","interfaces":[{"interface":"back","average":{"1min":17.667999999999999,"5min":40.302999999999997,"15min":26.126999999999999},"min":{"1min":10.754,"5min":7.0670000000000002,"15min":6.665},"max":{"1min":34.997,"5min":385.50799999999998,"15min":385.50799999999998},"last":19.373999999999999},{"interface":"front","average":{"1min":21.75,"5min":42.832999999999998,"15min":28.548999999999999},"min":{"1min":11.871,"5min":9.6639999999999997,"15min":6.5839999999999996},"max":{"1min":47.709000000000003,"5min":337.73700000000002,"15min":337.73700000000002},"last":17.199000000000002}]},{"osd":6,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.641999999999999,"5min":39.985999999999997,"15min":31.425000000000001},"min":{"1min":13.096,"5min":10.368,"15min":10.368},"max":{"1min":36.189999999999998,"5min":236.387,"15min":248.78899999999999},"last":11.795},{"interface":"front","average":{"1min":17.960999999999999,"5min":34.942,"15min":28.195},"min":{"1min":7.9359999999999999,"5min":6.4889999999999999,"15min":6.4889999999999999},"max":{"1min":88.625,"5min":252.99700000000001,"15min":381.90800000000002},"last":14.997999999999999}]},{"osd":7,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.481000000000002,"5min":32.908000000000001,"15min":28.715},"min":{"1min":8.6880000000000006,"5min":8.6880000000000006,"15min":8.4670000000000005},"max":{"1min":56.813000000000002,"5min":232.21600000000001,"15min":236.81299999999999},"last":12.441000000000001},{"interface":"front","average":{"1min":21.489999999999998,"5min":37.271000000000001,"15min":28.934000000000001},"min":{"1min":12.420999999999999,"5min":10.638,"15min":7.2409999999999997},"max":{"1min":34.862000000000002,"5min":249.29499999999999,"15min":249.29499999999999},"last":13.699999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509882,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72540,"kb_used_data":5264,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299300,"statfs":{"total":96636764160,"available":96562483200,"internally_reserved":0,"allocated":5390336,"data_stored":2682863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":21.77,"5min":23.620999999999999,"15min":18.902000000000001},"min":{"1min":5.0140000000000002,"5min":4.4569999999999999,"15min":4.3319999999999999},"max":{"1min":203.28899999999999,"5min":232.74199999999999,"15min":278.56900000000002},"last":5.2510000000000003},{"interface":"front","average":{"1min":19.181999999999999,"5min":22.715,"15min":17.931000000000001},"min":{"1min":7.4649999999999999,"5min":5.4530000000000003,"15min":4.8129999999999997},"max":{"1min":115.354,"5min":234.15799999999999,"15min":305.03800000000001},"last":9.2330000000000005}]},{"osd":1,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":31.167000000000002,"5min":26.742000000000001,"15min":23.835999999999999},"min":{"1min":7.3579999999999997,"5min":6.0549999999999997,"15min":6.0549999999999997},"max":{"1min":192.631,"5min":227.602,"15min":304.28300000000002},"last":14.456},{"interface":"front","average":{"1min":28.222000000000001,"5min":23.238,"15min":20.527000000000001},"min":{"1min":6.5300000000000002,"5min":6.5300000000000002,"15min":5.0199999999999996},"max":{"1min":175.59700000000001,"5min":175.59700000000001,"15min":224.27199999999999},"last":8.3580000000000005}]},{"osd":2,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":23.504999999999999,"5min":30.835000000000001,"15min":29.007000000000001},"min":{"1min":6.375,"5min":6.375,"15min":6.375},"max":{"1min":105.164,"5min":239.91900000000001,"15min":311.17200000000003},"last":17.838000000000001},{"interface":"front","average":{"1min":22.550000000000001,"5min":26.795000000000002,"15min":25.391999999999999},"min":{"1min":7.056,"5min":6.8170000000000002,"15min":6.8170000000000002},"max":{"1min":104.66500000000001,"5min":209.00899999999999,"15min":314.78699999999998},"last":11.32}]},{"osd":3,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":26.381,"5min":25.306000000000001,"15min":20.504000000000001},"min":{"1min":7.7729999999999997,"5min":6.7690000000000001,"15min":6.274},"max":{"1min":179.31299999999999,"5min":187.23099999999999,"15min":274.71899999999999},"last":13.635},{"interface":"front","average":{"1min":24.838000000000001,"5min":28.155999999999999,"15min":24.609999999999999},"min":{"1min":11.1,"5min":8.0670000000000002,"15min":7.3819999999999997},"max":{"1min":108.70099999999999,"5min":175.44800000000001,"15min":233.72800000000001},"last":12.835000000000001}]},{"osd":4,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":24.469000000000001,"5min":26.721,"15min":22.181000000000001},"min":{"1min":8.5410000000000004,"5min":6.4269999999999996,"15min":5.9059999999999997},"max":{"1min":132.90299999999999,"5min":185.755,"15min":219.21899999999999},"last":9.9589999999999996},{"interface":"front","average":{"1min":26.553000000000001,"5min":30.170999999999999,"15min":25.722000000000001},"min":{"1min":9.5739999999999998,"5min":8.7789999999999999,"15min":7.6689999999999996},"max":{"1min":116.852,"5min":174.81,"15min":174.81},"last":15.851000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":25.891999999999999,"5min":31.309000000000001,"15min":27.731999999999999},"min":{"1min":10.345000000000001,"5min":6.9299999999999997,"15min":6.9299999999999997},"max":{"1min":96.703999999999994,"5min":234.95099999999999,"15min":259.803},"last":18.640999999999998},{"interface":"front","average":{"1min":30.484000000000002,"5min":33.567999999999998,"15min":29.574999999999999},"min":{"1min":10.73,"5min":10.73,"15min":8.2859999999999996},"max":{"1min":98.491,"5min":206.55799999999999,"15min":206.55799999999999},"last":17.457999999999998}]},{"osd":7,"last update":"Sat Jan 27 04:20:14 2024","interfaces":[{"interface":"back","average":{"1min":33.548999999999999,"5min":31.448,"15min":26.803000000000001},"min":{"1min":11.276999999999999,"5min":8.0739999999999998,"15min":8.0739999999999998},"max":{"1min":180.07300000000001,"5min":190.39099999999999,"15min":319.75400000000002},"last":20.158999999999999},{"interface":"front","average":{"1min":32.110999999999997,"5min":34.119999999999997,"15min":28.417999999999999},"min":{"1min":10.561999999999999,"5min":9.0800000000000001,"15min":6.6459999999999999},"max":{"1min":217.697,"5min":240.69800000000001,"15min":277.59500000000003},"last":15.109}]}]},{"osd":7,"up_from":13,"seq":55834575291,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72164,"kb_used_data":4888,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299676,"statfs":{"total":96636764160,"available":96562868224,"internally_reserved":0,"allocated":5005312,"data_stored":2288496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:00 2024","interfaces":[{"interface":"back","average":{"1min":10.483000000000001,"5min":21.335000000000001,"15min":15.542999999999999},"min":{"1min":5.649,"5min":4.7370000000000001,"15min":4.1779999999999999},"max":{"1min":24.736999999999998,"5min":162.80799999999999,"15min":162.80799999999999},"last":6.8440000000000003},{"interface":"front","average":{"1min":13.324,"5min":28.123999999999999,"15min":19.064},"min":{"1min":6.5460000000000003,"5min":4.4020000000000001,"15min":4.4020000000000001},"max":{"1min":22.442,"5min":265.62200000000001,"15min":265.62200000000001},"last":11.18}]},{"osd":1,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":14.460000000000001,"5min":25.378,"15min":21.184000000000001},"min":{"1min":9.5470000000000006,"5min":6.0899999999999999,"15min":6.0899999999999999},"max":{"1min":22.472999999999999,"5min":171.16800000000001,"15min":204.17099999999999},"last":11.798999999999999},{"interface":"front","average":{"1min":8.282,"5min":20.821000000000002,"15min":18.806999999999999},"min":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"max":{"1min":20.916,"5min":192.11000000000001,"15min":323.48599999999999},"last":14.613}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.647,"5min":30.683,"15min":21.582999999999998},"min":{"1min":9.0679999999999996,"5min":6.1779999999999999,"15min":6.1269999999999998},"max":{"1min":23.294,"5min":420.65199999999999,"15min":420.65199999999999},"last":9.3480000000000008},{"interface":"front","average":{"1min":13.297000000000001,"5min":28.969000000000001,"15min":22.788},"min":{"1min":6.25,"5min":6.25,"15min":6.25},"max":{"1min":36.289000000000001,"5min":453.39100000000002,"15min":453.39100000000002},"last":13.087999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":14.026,"5min":27.434000000000001,"15min":23.035},"min":{"1min":9.8599999999999994,"5min":8.0830000000000002,"15min":7.6890000000000001},"max":{"1min":23.341999999999999,"5min":226.72300000000001,"15min":226.72300000000001},"last":8.4209999999999994},{"interface":"front","average":{"1min":15.734999999999999,"5min":24.466000000000001,"15min":21.619},"min":{"1min":7.6680000000000001,"5min":5.7270000000000003,"15min":5.7270000000000003},"max":{"1min":23.318999999999999,"5min":197.01499999999999,"15min":197.01499999999999},"last":12.484}]},{"osd":4,"last update":"Sat Jan 27 04:20:11 2024","interfaces":[{"interface":"back","average":{"1min":15.734,"5min":29.295000000000002,"15min":22.515000000000001},"min":{"1min":9.2780000000000005,"5min":8.4749999999999996,"15min":8.3200000000000003},"max":{"1min":48.780000000000001,"5min":227.72300000000001,"15min":227.72300000000001},"last":10.061999999999999},{"interface":"front","average":{"1min":18.984000000000002,"5min":32.296999999999997,"15min":24.923999999999999},"min":{"1min":11.102,"5min":9.3059999999999992,"15min":9.3059999999999992},"max":{"1min":55.207000000000001,"5min":230.16,"15min":230.16},"last":18.850000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:19:22 2024","interfaces":[{"interface":"back","average":{"1min":18.823,"5min":38.701000000000001,"15min":28.065000000000001},"min":{"1min":14.417,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":49.390999999999998,"5min":276.23500000000001,"15min":276.23500000000001},"last":13.885},{"interface":"front","average":{"1min":21.015999999999998,"5min":35.655000000000001,"15min":29.832999999999998},"min":{"1min":16.212,"5min":10.522,"15min":10.522},"max":{"1min":36.661000000000001,"5min":408.774,"15min":408.774},"last":18.050000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":20.045999999999999,"5min":34.045000000000002,"15min":29.82},"min":{"1min":15.138,"5min":10.936,"15min":10.936},"max":{"1min":27.131,"5min":276.92099999999999,"15min":304.952},"last":17.254000000000001},{"interface":"front","average":{"1min":14.42,"5min":30.312000000000001,"15min":26.094000000000001},"min":{"1min":7.2199999999999998,"5min":7.2199999999999998,"15min":7.2199999999999998},"max":{"1min":25.401,"5min":302.61700000000002,"15min":302.61700000000002},"last":15.374000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T04:20:19.387 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-27T04:20:19.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.488+0000 7ff4b18b4640 10 monclient: get_monmap_and_config 2024-01-27T04:20:19.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.488+0000 7ff4b18b4640 10 monclient: build_initial_monmap 2024-01-27T04:20:19.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.488+0000 7ff4b18b4640 10 monclient: monmap: 2024-01-27T04:20:19.489 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:19.489359+0000 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:19.489359+0000 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:19.490 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:19.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.488+0000 7ff4b18b4640 1 Processor -- start 2024-01-27T04:20:19.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.488+0000 7ff4b18b4640 1 -- start start 2024-01-27T04:20:19.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.488+0000 7ff4b18b4640 10 monclient: init 2024-01-27T04:20:19.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:19.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac0b4680 0x7ff4ac142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 10 monclient(hunting): picked mon.noname-b con 0x7ff4ac0b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4ac0b4a60 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4ac142eb0 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ac0afbe0 con 0x7ff4ac142eb0 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ac0b0890 con 0x7ff4ac0b4a60 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac142240 con 0x7ff4ac0b4680 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4b18b4640 10 monclient(hunting): authenticate will time out at 3702.843364s 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.489+0000 7ff4aaffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac0b4680 0x7ff4ac142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.490+0000 7ff4ab7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff4ac142eb0 0x7ff4ac14caf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48060/0 (socket says 172.21.15.110:48060) 2024-01-27T04:20:19.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.490+0000 7ff4ab7fe640 1 -- 172.21.15.110:0/2503724165 learned_addr learned my addr 172.21.15.110:0/2503724165 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T04:20:19.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.490+0000 7ff4aaffd640 10 monclient(hunting): get_auth_request con 0x7ff4ac0b4680 auth_method 0 2024-01-27T04:20:19.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.490+0000 7ff4aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:19.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.490+0000 7ff4aaffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:19.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.490+0000 7ff4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:19.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.490+0000 7ff4aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:19.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.490+0000 7ff4aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:19.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.490+0000 7ff4aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:19.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 1 -- 172.21.15.110:0/2503724165 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463905503 0 0) 0x7ff4ac0b0890 con 0x7ff4ac0b4a60 2024-01-27T04:20:19.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 10 monclient(hunting): _negotiate 2024-01-27T04:20:19.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:19.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:19.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 10 monclient(hunting): my global_id is 20896 2024-01-27T04:20:19.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 1 -- 172.21.15.110:0/2503724165 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff494003520 con 0x7ff4ac0b4a60 2024-01-27T04:20:19.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 1 -- 172.21.15.110:0/2503724165 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023211230 0 0) 0x7ff4ac0afbe0 con 0x7ff4ac142eb0 2024-01-27T04:20:19.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 10 monclient(hunting): _negotiate 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 10 monclient(hunting): my global_id is 16989 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4a9ffb640 1 -- 172.21.15.110:0/2503724165 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ac0b0890 con 0x7ff4ac142eb0 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4aaffd640 10 monclient(hunting): handle_auth_done global_id 18863 payload 274 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4aaffd640 1 -- 172.21.15.110:0/2503724165 >> v1:172.21.15.162:6789/0 conn(0x7ff4ac0b4a60 legacy=0x7ff4ac0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4aaffd640 1 -- 172.21.15.110:0/2503724165 >> v1:172.21.15.110:6789/0 conn(0x7ff4ac142eb0 legacy=0x7ff4ac14caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4aaffd640 10 monclient: _finish_hunting 0 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4aaffd640 1 monclient: found mon.noname-b 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4aaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.492+0000 7ff4aaffd640 1 -- 172.21.15.110:0/2503724165 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac0b0b00 con 0x7ff4ac0b4680 2024-01-27T04:20:19.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.494+0000 7ff4aaffd640 1 --2- 172.21.15.110:0/2503724165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac0b4680 0x7ff4ac142970 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7ff49c0097d0 tx=0x7ff49c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56da5f6ad9f4b59e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:19.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 1 -- 172.21.15.110:0/2503724165 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c031800 con 0x7ff4ac0b4680 2024-01-27T04:20:19.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:19.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:19.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 10 monclient: dump: 2024-01-27T04:20:19.499 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:19.499 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:19.499 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:19.499 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:19.499 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:19.500 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:19.500 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:19.500 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:19.500 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:19.500 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 10 monclient: _finish_auth 0 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 10 monclient: _check_auth_tickets 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 1 -- 172.21.15.110:0/2503724165 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49c031960 con 0x7ff4ac0b4680 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 1 -- 172.21.15.110:0/2503724165 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c031cf0 con 0x7ff4ac0b4680 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 5 monclient: authenticate success, global_id 18863 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 4 set_mon_vals no callback set 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:19.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4a9ffb640 10 monclient: dump: 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:19.502 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:19.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 10 monclient: shutdown 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2503724165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac0b4680 msgr2=0x7ff4ac142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 1 --2- 172.21.15.110:0/2503724165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac0b4680 0x7ff4ac142970 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7ff49c0097d0 tx=0x7ff49c02b8a0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2503724165 shutdown_connections 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 1 --2- 172.21.15.110:0/2503724165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac0b4680 0x7ff4ac142970 unknown :-1 s=CLOSED pgs=3408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2503724165 >> 172.21.15.110:0/2503724165 conn(0x7ff4ac066490 msgr2=0x7ff4ac0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2503724165 shutdown_connections 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.498+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2503724165 wait complete. 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.499+0000 7ff4b18b4640 10 monclient: build_initial_monmap 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.499+0000 7ff4b18b4640 10 monclient: monmap: 2024-01-27T04:20:19.508 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.499+0000 7ff4b18b4640 1 librados: starting msgr at 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.499+0000 7ff4b18b4640 1 librados: starting objecter 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.499+0000 7ff4b18b4640 1 Processor -- start 2024-01-27T04:20:19.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 1 -- start start 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 1 librados: setting wanted keys 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 1 librados: calling monclient init 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient: init 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4ac0b4680 0x7ff4ac0aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient(hunting): picked mon.noname-b con 0x7ff4ac0b4680 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac142eb0 0x7ff4ac0ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient(hunting): picked mon.noname-c con 0x7ff4ac142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4ac1da760 0x7ff4ac1dabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient(hunting): picked mon.noname-a con 0x7ff4ac1da760 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:19.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0aba20 con 0x7ff4ac1da760 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0abcd0 con 0x7ff4ac0b4680 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac1459e0 con 0x7ff4ac142eb0 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.500+0000 7ff4b18b4640 10 monclient(hunting): authenticate will time out at 3702.854565s 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aa7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac142eb0 0x7ff4ac0ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4ac0b4680 0x7ff4ac0aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4ab7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4ac1da760 0x7ff4ac1dabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:19.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4ac0b4680 0x7ff4ac0aaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:58774/0 (socket says 172.21.15.110:58774) 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aaffd640 1 -- 172.21.15.110:0/2506311080 learned_addr learned my addr 172.21.15.110:0/2506311080 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aaffd640 10 monclient(hunting): get_auth_request con 0x7ff4ac0b4680 auth_method 0 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aaffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aa7fc640 10 monclient(hunting): get_auth_request con 0x7ff4ac142eb0 auth_method 0 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aa7fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4ab7fe640 10 monclient(hunting): get_auth_request con 0x7ff4ac1da760 auth_method 0 2024-01-27T04:20:19.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4ab7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.501+0000 7ff4ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aa7fc640 1 -- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac142eb0 msgr2=0x7ff4ac0ab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-01-27T04:20:19.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aa7fc640 1 -- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac142eb0 msgr2=0x7ff4ac0ab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aa7fc640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac142eb0 0x7ff4ac0ab4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.502+0000 7ff4aa7fc640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac142eb0 0x7ff4ac0ab4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 10 monclient(hunting): handle_auth_done global_id 20899 payload 824 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 1 -- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac142eb0 msgr2=0x7ff4ac0ab4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac142eb0 0x7ff4ac0ab4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 1 -- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4ac1da760 msgr2=0x7ff4ac1dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4ac1da760 0x7ff4ac1dabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 10 monclient: _finish_hunting 0 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 1 monclient: found mon.noname-b 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:19.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:19.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.504+0000 7ff4aaffd640 1 -- 172.21.15.110:0/2506311080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c009400 con 0x7ff4ac0b4680 2024-01-27T04:20:19.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.506+0000 7ff4aaffd640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4ac0b4680 0x7ff4ac0aaeb0 secure :-1 s=READY pgs=5083 cs=0 l=1 rev1=1 crypto rx=0x7ff49c0098f0 tx=0x7ff49c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:19.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.506+0000 7ff493fff640 10 client.?.objecter ms_handle_connect 0x7ff4ac0b4680 2024-01-27T04:20:19.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.506+0000 7ff493fff640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:19.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 1 -- 172.21.15.110:0/2506311080 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c03bc70 con 0x7ff4ac0b4680 2024-01-27T04:20:19.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 10 monclient: dump: 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:19.517 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 10 monclient: _finish_auth 0 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 10 monclient: _check_auth_tickets 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 1 -- 172.21.15.110:0/2506311080 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49c03bdd0 con 0x7ff4ac0b4680 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 1 -- 172.21.15.110:0/2506311080 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c045630 con 0x7ff4ac0b4680 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff493fff640 10 monclient: dump: 2024-01-27T04:20:19.518 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4b18b4640 5 monclient: authenticate success, global_id 20899 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4a97fa640 4 set_mon_vals no callback set 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4b18b4640 10 monclient: _renew_subs 2024-01-27T04:20:19.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4b18b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ac1db130 con 0x7ff4ac0b4680 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4b18b4640 10 client.20899.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4b18b4640 10 monclient: _renew_subs 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4b18b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac13ab60 con 0x7ff4ac0b4680 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.510+0000 7ff4b18b4640 1 librados: init done 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.511+0000 7ff4b18b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.511+0000 7ff4b18b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.511+0000 7ff4b18b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:19.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.511+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac145c00 con 0x7ff4ac0b4680 2024-01-27T04:20:19.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.529+0000 7ff493fff640 1 -- 172.21.15.110:0/2506311080 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7ff49c04e450 con 0x7ff4ac0b4680 2024-01-27T04:20:19.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.529+0000 7ff493fff640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff49806c9f0 0x7ff49806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:19.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.530+0000 7ff4aa7fc640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff49806c9f0 0x7ff49806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:19.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.530+0000 7ff4aa7fc640 10 monclient: get_auth_request con 0x7ff49806c9f0 auth_method 0 2024-01-27T04:20:19.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.530+0000 7ff4aa7fc640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff49806c9f0 0x7ff49806eeb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0aabf0 tx=0x7ff4a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:19.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.530+0000 7ff493fff640 10 client.20899.objecter ms_handle_connect 0x7ff49806c9f0 2024-01-27T04:20:19.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.532+0000 7ff493fff640 1 -- 172.21.15.110:0/2506311080 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7ff49c02e070 con 0x7ff4ac0b4680 2024-01-27T04:20:19.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.532+0000 7ff493fff640 10 client.20899.objecter ms_dispatch 0x7ff4ac000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:19.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.532+0000 7ff493fff640 3 client.20899.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:19.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.532+0000 7ff493fff640 3 client.20899.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:19.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.532+0000 7ff493fff640 20 client.20899.objecter dump_active .. 0 homeless 2024-01-27T04:20:19.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.767+0000 7ff493fff640 1 -- 172.21.15.110:0/2506311080 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7ff49c076e50 con 0x7ff4ac0b4680 2024-01-27T04:20:19.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.767+0000 7ff493fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:19.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.767+0000 7ff493fff640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:19.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.941+0000 7ff4b18b4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-27T04:20:19.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.941+0000 7ff4b18b4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T04:20:19.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.941+0000 7ff4b18b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:19.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.941+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4ac074d10 con 0x7ff4ac0b4680 2024-01-27T04:20:19.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.978+0000 7ff493fff640 1 -- 172.21.15.110:0/2506311080 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1334) v1 ==== 74+0+17704 (secure 0 0 0) 0x7ff49c035020 con 0x7ff4ac0b4680 2024-01-27T04:20:19.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.978+0000 7ff493fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-27T04:20:19.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.978+0000 7ff493fff640 10 monclient: _finish_command 2 = system:0 2024-01-27T04:20:19.980 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T04:20:19.980 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":1334,"fsid":"23f901d8-28a2-405b-9f8e-3221d4911c2c","created":"2024-01-27T03:42:25.256331+0000","modified":"2024-01-27T04:20:16.597946+0000","last_up_change":"2024-01-27T03:43:21.487334+0000","last_in_change":"2024-01-27T03:42:30.608380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":115,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-27T03:43:13.074487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-27T03:43:27.987244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-01-27T03:44:50.741596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"112d1665-ee1b-41ea-85fc-4165aa26818e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1330,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6817","nonce":1545415171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6819","nonce":1545415171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6824","nonce":1545415171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":1545415171},{"type":"v1","addr":"172.21.15.110:6821","nonce":1545415171}]},"public_addr":"172.21.15.110:6817/1545415171","cluster_addr":"172.21.15.110:6819/1545415171","heartbeat_back_addr":"172.21.15.110:6824/1545415171","heartbeat_front_addr":"172.21.15.110:6821/1545415171","state":["exists","up"]},{"osd":1,"uuid":"1c7f3c75-90ac-4a98-bc89-b6bd97b20f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":1314,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6823","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6825","nonce":518896551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6827","nonce":518896551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6831","nonce":518896551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":518896551},{"type":"v1","addr":"172.21.15.110:6829","nonce":518896551}]},"public_addr":"172.21.15.110:6825/518896551","cluster_addr":"172.21.15.110:6827/518896551","heartbeat_back_addr":"172.21.15.110:6831/518896551","heartbeat_front_addr":"172.21.15.110:6829/518896551","state":["exists","up"]},{"osd":2,"uuid":"5d9d24ee-f76a-47e1-86c8-6ec8e22805bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":1322,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6809","nonce":3683566991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6811","nonce":3683566991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6815","nonce":3683566991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3683566991},{"type":"v1","addr":"172.21.15.110:6813","nonce":3683566991}]},"public_addr":"172.21.15.110:6809/3683566991","cluster_addr":"172.21.15.110:6811/3683566991","heartbeat_back_addr":"172.21.15.110:6815/3683566991","heartbeat_front_addr":"172.21.15.110:6813/3683566991","state":["exists","up"]},{"osd":3,"uuid":"2b5190b6-abce-4528-9ea6-8363bdd89328","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1330,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6801","nonce":2374111370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6803","nonce":2374111370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6807","nonce":2374111370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2374111370},{"type":"v1","addr":"172.21.15.110:6805","nonce":2374111370}]},"public_addr":"172.21.15.110:6801/2374111370","cluster_addr":"172.21.15.110:6803/2374111370","heartbeat_back_addr":"172.21.15.110:6807/2374111370","heartbeat_front_addr":"172.21.15.110:6805/2374111370","state":["exists","up"]},{"osd":4,"uuid":"ea281cce-cc93-4030-adbe-ee9e16296537","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1330,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6827","nonce":1007517023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6829","nonce":1007517023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6833","nonce":1007517023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1007517023},{"type":"v1","addr":"172.21.15.162:6831","nonce":1007517023}]},"public_addr":"172.21.15.162:6827/1007517023","cluster_addr":"172.21.15.162:6829/1007517023","heartbeat_back_addr":"172.21.15.162:6833/1007517023","heartbeat_front_addr":"172.21.15.162:6831/1007517023","state":["exists","up"]},{"osd":5,"uuid":"36bd4a7f-c1c8-4c8b-aea3-994fdf4dfe47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1330,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6803","nonce":946981202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6805","nonce":946981202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6809","nonce":946981202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":946981202},{"type":"v1","addr":"172.21.15.162:6807","nonce":946981202}]},"public_addr":"172.21.15.162:6803/946981202","cluster_addr":"172.21.15.162:6805/946981202","heartbeat_back_addr":"172.21.15.162:6809/946981202","heartbeat_front_addr":"172.21.15.162:6807/946981202","state":["exists","up"]},{"osd":6,"uuid":"26b0b3bd-a412-4d75-a5ab-86df55753f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1330,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6811","nonce":3628034141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6813","nonce":3628034141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6817","nonce":3628034141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3628034141},{"type":"v1","addr":"172.21.15.162:6815","nonce":3628034141}]},"public_addr":"172.21.15.162:6811/3628034141","cluster_addr":"172.21.15.162:6813/3628034141","heartbeat_back_addr":"172.21.15.162:6817/3628034141","heartbeat_front_addr":"172.21.15.162:6815/3628034141","state":["exists","up"]},{"osd":7,"uuid":"7b8b531d-33ed-4bd5-8b97-d5ff4b210c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1330,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6819","nonce":68785612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6821","nonce":68785612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6825","nonce":68785612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":68785612},{"type":"v1","addr":"172.21.15.162:6823","nonce":68785612}]},"public_addr":"172.21.15.162:6819/68785612","cluster_addr":"172.21.15.162:6821/68785612","heartbeat_back_addr":"172.21.15.162:6825/68785612","heartbeat_front_addr":"172.21.15.162:6823/68785612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.313498+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:10.279187+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.161615+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.904665+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:08.256516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:09.104354+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:08.722052+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-27T03:43:07.959074+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":10,"snaps":[{"begin":5,"length":2}]},{"pool":21,"snaps":[{"begin":1,"length":1}]},{"pool":24,"snaps":[{"begin":2,"length":3}]},{"pool":66,"snaps":[{"begin":1,"length":1}]},{"pool":89,"snaps":[{"begin":2,"length":4},{"begin":7,"length":2}]},{"pool":98,"snaps":[{"begin":1,"length":1}]},{"pool":102,"snaps":[{"begin":1,"length":2}]},{"pool":105,"snaps":[{"begin":3,"length":2}]},{"pool":107,"snaps":[{"begin":3,"length":1}]},{"pool":120,"snaps":[{"begin":2,"length":2}]},{"pool":133,"snaps":[{"begin":4,"length":2}]},{"pool":134,"snaps":[{"begin":2,"length":3}]},{"pool":141,"snaps":[{"begin":2,"length":2}]},{"pool":157,"snaps":[{"begin":2,"length":1}]},{"pool":158,"snaps":[{"begin":2,"length":1}]},{"pool":161,"snaps":[{"begin":1,"length":4}]},{"pool":168,"snaps":[{"begin":4,"length":4}]},{"pool":169,"snaps":[{"begin":3,"length":4}]},{"pool":195,"snaps":[{"begin":7,"length":1}]},{"pool":201,"snaps":[{"begin":8,"length":1}]},{"pool":203,"snaps":[{"begin":10,"length":1}]},{"pool":204,"snaps":[{"begin":11,"length":2}]},{"pool":207,"snaps":[{"begin":14,"length":1}]},{"pool":250,"snaps":[{"begin":2,"length":1}]},{"pool":251,"snaps":[{"begin":3,"length":1}]},{"pool":254,"snaps":[{"begin":4,"length":1}]},{"pool":258,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-27T04:20:19.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 10 librados: watch_flush enter 2024-01-27T04:20:19.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 10 librados: watch_flush exit 2024-01-27T04:20:19.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 20 client.20899.objecter shutdown clearing up homeless session... 2024-01-27T04:20:19.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 10 client.20899.objecter successfully canceled tick 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff49806c9f0 msgr2=0x7ff49806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff49806c9f0 0x7ff49806eeb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0aabf0 tx=0x7ff4a0009040 comp rx=0 tx=0).stop 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 10 monclient: shutdown 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4ac0b4680 msgr2=0x7ff4ac0aaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4ac0b4680 0x7ff4ac0aaeb0 secure :-1 s=READY pgs=5083 cs=0 l=1 rev1=1 crypto rx=0x7ff49c0098f0 tx=0x7ff49c0046c0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 shutdown_connections 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff49806c9f0 0x7ff49806eeb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4ac142eb0 0x7ff4ac0ab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4ac1da760 0x7ff4ac1dabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 --2- 172.21.15.110:0/2506311080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4ac0b4680 0x7ff4ac0aaeb0 unknown :-1 s=CLOSED pgs=5083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:19.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 >> 172.21.15.110:0/2506311080 conn(0x7ff4ac066490 msgr2=0x7ff4ac0b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:19.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 shutdown_connections 2024-01-27T04:20:19.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.980+0000 7ff4b18b4640 1 -- 172.21.15.110:0/2506311080 wait complete. 2024-01-27T04:20:19.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:19.981+0000 7ff4b18b4640 1 librados: shutdown 2024-01-27T04:20:20.995 INFO:tasks.ceph:Scrubbing osd.0 2024-01-27T04:20:20.995 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.097+0000 7ff64b2b4640 10 monclient: get_monmap_and_config 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.097+0000 7ff64b2b4640 10 monclient: build_initial_monmap 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.097+0000 7ff64b2b4640 10 monclient: monmap: 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:21.098185+0000 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:21.098185+0000 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:21.098 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:21.099 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:21.099 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:21.099 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:21.099 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.097+0000 7ff64b2b4640 1 Processor -- start 2024-01-27T04:20:21.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.097+0000 7ff64b2b4640 1 -- start start 2024-01-27T04:20:21.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.097+0000 7ff64b2b4640 10 monclient: init 2024-01-27T04:20:21.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:21.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff64406a940 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff644142de0 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff6440a9e40 0x7ff6440b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 10 monclient(hunting): picked mon.noname-a con 0x7ff6440a9e40 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6440aac50 con 0x7ff64406a940 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6440aaf30 con 0x7ff644142de0 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64406ad20 con 0x7ff6440a9e40 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff64b2b4640 10 monclient(hunting): authenticate will time out at 3704.452387s 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.098+0000 7ff649851640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff6440a9e40 0x7ff6440b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.099+0000 7ff64884f640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ff644142de0 0x7ff6440b2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:34566/0 (socket says 172.21.15.110:34566) 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.099+0000 7ff64884f640 1 -- 172.21.15.110:0/2316533472 learned_addr learned my addr 172.21.15.110:0/2316533472 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T04:20:21.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.099+0000 7ff649851640 10 monclient(hunting): get_auth_request con 0x7ff6440a9e40 auth_method 0 2024-01-27T04:20:21.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.099+0000 7ff649851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.099+0000 7ff649851640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.099+0000 7ff649851640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.100+0000 7ff649851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.100+0000 7ff649851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.100+0000 7ff649851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.101+0000 7ff63bfff640 1 -- 172.21.15.110:0/2316533472 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254715717 0 0) 0x7ff6440aaf30 con 0x7ff644142de0 2024-01-27T04:20:21.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.101+0000 7ff63bfff640 10 monclient(hunting): _negotiate 2024-01-27T04:20:21.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.101+0000 7ff63bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.101+0000 7ff63bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.101+0000 7ff63bfff640 10 monclient(hunting): my global_id is 20902 2024-01-27T04:20:21.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.101+0000 7ff63bfff640 1 -- 172.21.15.110:0/2316533472 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff630003520 con 0x7ff644142de0 2024-01-27T04:20:21.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.102+0000 7ff649851640 10 monclient(hunting): handle_auth_done global_id 16995 payload 274 2024-01-27T04:20:21.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.102+0000 7ff649851640 1 -- 172.21.15.110:0/2316533472 >> v1:172.21.15.162:6789/0 conn(0x7ff644142de0 legacy=0x7ff6440b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.102+0000 7ff649851640 1 -- 172.21.15.110:0/2316533472 >> v1:172.21.15.110:6789/0 conn(0x7ff64406a940 legacy=0x7ff6441426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.102+0000 7ff649851640 10 monclient: _finish_hunting 0 2024-01-27T04:20:21.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.102+0000 7ff649851640 1 monclient: found mon.noname-a 2024-01-27T04:20:21.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.102+0000 7ff649851640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:21.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.102+0000 7ff649851640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T04:20:21.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.102+0000 7ff649851640 1 -- 172.21.15.110:0/2316533472 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6441421b0 con 0x7ff6440a9e40 2024-01-27T04:20:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.107+0000 7ff649851640 1 --2- 172.21.15.110:0/2316533472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6440a9e40 0x7ff6440b3450 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0097a0 tx=0x7ff63c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cea0a6f9f00c0e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 1 -- 172.21.15.110:0/2316533472 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c030610 con 0x7ff6440a9e40 2024-01-27T04:20:21.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T04:20:21.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 10 monclient: dump: 2024-01-27T04:20:21.111 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.111 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.111 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 10 monclient: _finish_auth 0 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 10 monclient: _check_auth_tickets 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 1 -- 172.21.15.110:0/2316533472 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c030770 con 0x7ff6440a9e40 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 1 -- 172.21.15.110:0/2316533472 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c030b00 con 0x7ff6440a9e40 2024-01-27T04:20:21.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff64b2b4640 5 monclient: authenticate success, global_id 16995 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff64b2b4640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff64b2b4640 4 set_mon_vals no callback set 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff63bfff640 10 monclient: dump: 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.113 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff64b2b4640 10 monclient: shutdown 2024-01-27T04:20:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff64b2b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff64b2b4640 1 -- 172.21.15.110:0/2316533472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6440a9e40 msgr2=0x7ff6440b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.110+0000 7ff64b2b4640 1 --2- 172.21.15.110:0/2316533472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6440a9e40 0x7ff6440b3450 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0097a0 tx=0x7ff63c02b960 comp rx=0 tx=0).stop 2024-01-27T04:20:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.111+0000 7ff64b2b4640 1 -- 172.21.15.110:0/2316533472 shutdown_connections 2024-01-27T04:20:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.111+0000 7ff64b2b4640 1 --2- 172.21.15.110:0/2316533472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6440a9e40 0x7ff6440b3450 unknown :-1 s=CLOSED pgs=3463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.111+0000 7ff64b2b4640 1 -- 172.21.15.110:0/2316533472 >> 172.21.15.110:0/2316533472 conn(0x7ff6440b7d70 msgr2=0x7ff6440b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.111+0000 7ff64b2b4640 1 -- 172.21.15.110:0/2316533472 shutdown_connections 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.111+0000 7ff64b2b4640 1 -- 172.21.15.110:0/2316533472 wait complete. 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.111+0000 7ff64b2b4640 10 monclient: build_initial_monmap 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.111+0000 7ff64b2b4640 10 monclient: monmap: 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 librados: starting msgr at 2024-01-27T04:20:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 librados: starting objecter 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 Processor -- start 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 -- start start 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 librados: setting wanted keys 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 librados: calling monclient init 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient: init 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64406a940 0x7ff64413e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient(hunting): picked mon.noname-a con 0x7ff64406a940 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6440a9e40 0x7ff6440ac350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient(hunting): picked mon.noname-b con 0x7ff6440a9e40 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff644142de0 0x7ff64413c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient(hunting): picked mon.noname-c con 0x7ff644142de0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6440ac890 con 0x7ff64406a940 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6440ab970 con 0x7ff6440a9e40 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6440abc50 con 0x7ff644142de0 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.112+0000 7ff64b2b4640 10 monclient(hunting): authenticate will time out at 3704.466787s 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649050640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64406a940 0x7ff64413e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649851640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff644142de0 0x7ff64413c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff64884f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6440a9e40 0x7ff6440ac350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649050640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64406a940 0x7ff64413e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44888/0 (socket says 172.21.15.110:44888) 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649050640 1 -- 172.21.15.110:0/3266086415 learned_addr learned my addr 172.21.15.110:0/3266086415 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649050640 10 monclient(hunting): get_auth_request con 0x7ff64406a940 auth_method 0 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649050640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649050640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649050640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff64884f640 10 monclient(hunting): get_auth_request con 0x7ff6440a9e40 auth_method 0 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff64884f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff64884f640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff64884f640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649851640 10 monclient(hunting): get_auth_request con 0x7ff644142de0 auth_method 0 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649851640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.113+0000 7ff649851640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.114+0000 7ff649050640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.114+0000 7ff649050640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.114+0000 7ff649050640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.114+0000 7ff64884f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.114+0000 7ff64884f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.114+0000 7ff64884f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.114+0000 7ff649851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.114+0000 7ff649851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.114+0000 7ff649851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 10 monclient(hunting): handle_auth_done global_id 17001 payload 824 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 1 -- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff644142de0 msgr2=0x7ff64413c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff644142de0 0x7ff64413c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 1 -- 172.21.15.110:0/3266086415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6440a9e40 msgr2=0x7ff6440ac350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6440a9e40 0x7ff6440ac350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 10 monclient: _finish_hunting 0 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 1 monclient: found mon.noname-a 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff649050640 1 -- 172.21.15.110:0/3266086415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c009400 con 0x7ff64406a940 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.116+0000 7ff64884f640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6440a9e40 0x7ff6440ac350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T04:20:21.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.119+0000 7ff649050640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64406a940 0x7ff64413e630 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7ff64000c730 tx=0x7ff64000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.119+0000 7ff639ffb640 10 client.?.objecter ms_handle_connect 0x7ff64406a940 2024-01-27T04:20:21.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.119+0000 7ff639ffb640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 1 -- 172.21.15.110:0/3266086415 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff640006d50 con 0x7ff64406a940 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 10 monclient: dump: 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.125 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 10 monclient: _finish_auth 0 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 10 monclient: _check_auth_tickets 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff64b2b4640 5 monclient: authenticate success, global_id 17001 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 1 -- 172.21.15.110:0/3266086415 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64000ce40 con 0x7ff64406a940 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 1 -- 172.21.15.110:0/3266086415 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff640003bb0 con 0x7ff64406a940 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff639ffb640 10 monclient: dump: 2024-01-27T04:20:21.126 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff63b7fe640 4 set_mon_vals no callback set 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff64b2b4640 10 monclient: _renew_subs 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff64b2b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6440abf30 con 0x7ff64406a940 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff64b2b4640 10 client.17001.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff64b2b4640 10 monclient: _renew_subs 2024-01-27T04:20:21.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff64b2b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:21.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64413d990 con 0x7ff64406a940 2024-01-27T04:20:21.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.122+0000 7ff64b2b4640 1 librados: init done 2024-01-27T04:20:21.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.123+0000 7ff64b2b4640 10 client.17001.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:21.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.123+0000 7ff64b2b4640 15 client.17001.objecter _session_command_op_assign -1 1 2024-01-27T04:20:21.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.123+0000 7ff64b2b4640 20 client.17001.objecter _get_session osd=-1 returning homeless 2024-01-27T04:20:21.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.123+0000 7ff64b2b4640 10 client.17001.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:21.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.123+0000 7ff64b2b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:21.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.123+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff64414da30 con 0x7ff64406a940 2024-01-27T04:20:21.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.142+0000 7ff639ffb640 1 -- 172.21.15.110:0/3266086415 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7ff640003d10 con 0x7ff64406a940 2024-01-27T04:20:21.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.142+0000 7ff639ffb640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff62806ca20 0x7ff62806eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.142+0000 7ff64884f640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff62806ca20 0x7ff62806eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.142+0000 7ff64884f640 10 monclient: get_auth_request con 0x7ff62806ca20 auth_method 0 2024-01-27T04:20:21.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.143+0000 7ff64884f640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff62806ca20 0x7ff62806eee0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff644146700 tx=0x7ff634007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.143+0000 7ff639ffb640 10 client.17001.objecter ms_handle_connect 0x7ff62806ca20 2024-01-27T04:20:21.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 1 -- 172.21.15.110:0/3266086415 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7ff640058bc0 con 0x7ff64406a940 2024-01-27T04:20:21.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 10 client.17001.objecter ms_dispatch 0x7ff644000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:21.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 3 client.17001.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 3 client.17001.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 10 client.17001.objecter checking command 1 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7ff6280747a0 0x7ff628076c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 20 client.17001.objecter _get_session s=0x7ff6280740d0 osd=0 3 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 20 client.17001.objecter put_session s=0x7ff6280740d0 osd=0 3 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 15 client.17001.objecter _session_command_op_remove -1 1 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 20 client.17001.objecter _get_session s=0x7ff6280740d0 osd=0 3 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 20 client.17001.objecter get_session s=0x7ff6280740d0 osd=0 3 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 15 client.17001.objecter _session_command_op_assign 0 1 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 20 client.17001.objecter put_session s=0x7ff6280740d0 osd=0 4 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 10 client.17001.objecter _send_command 1 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 1 -- 172.21.15.110:0/3266086415 --> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff628077310 con 0x7ff6280747a0 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.144+0000 7ff639ffb640 20 client.17001.objecter dump_active .. 0 homeless 2024-01-27T04:20:21.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.145+0000 7ff649851640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7ff6280747a0 0x7ff628076c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.145+0000 7ff649851640 10 monclient: get_auth_request con 0x7ff6280747a0 auth_method 0 2024-01-27T04:20:21.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.146+0000 7ff639ffb640 1 -- 172.21.15.110:0/3266086415 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=1334) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff640054e60 con 0x7ff64406a940 2024-01-27T04:20:21.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.146+0000 7ff639ffb640 10 monclient: handle_get_version_reply finishing 1 version 1334 2024-01-27T04:20:21.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.148+0000 7ff639ffb640 10 client.17001.objecter ms_handle_connect 0x7ff6280747a0 2024-01-27T04:20:21.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.148+0000 7ff649851640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7ff6280747a0 0x7ff628076c40 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.173+0000 7ff639ffb640 1 -- 172.21.15.110:0/3266086415 <== osd.0 v2:172.21.15.110:6816/1545415171 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff628077310 con 0x7ff6280747a0 2024-01-27T04:20:21.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.173+0000 7ff639ffb640 10 client.17001.objecter ms_dispatch 0x7ff644000b90 command_reply(tid 1: 0 ) v1 2024-01-27T04:20:21.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.173+0000 7ff639ffb640 10 client.17001.objecter _finish_command 1 = system:0 2024-01-27T04:20:21.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.173+0000 7ff639ffb640 20 client.17001.objecter put_session s=0x7ff6280740d0 osd=0 4 2024-01-27T04:20:21.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.173+0000 7ff639ffb640 15 client.17001.objecter _session_command_op_remove 0 1 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 10 client.17001.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 15 client.17001.objecter _session_command_op_assign -1 2 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 20 client.17001.objecter _get_session s=0x7ff6280740d0 osd=0 3 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 20 client.17001.objecter put_session s=0x7ff6280740d0 osd=0 3 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 20 client.17001.objecter _get_session s=0x7ff6280740d0 osd=0 3 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 15 client.17001.objecter _session_command_op_remove -1 2 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 20 client.17001.objecter get_session s=0x7ff6280740d0 osd=0 3 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 15 client.17001.objecter _session_command_op_assign 0 2 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 20 client.17001.objecter put_session s=0x7ff6280740d0 osd=0 4 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 10 client.17001.objecter _send_command 2 2024-01-27T04:20:21.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.187+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 --> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff64413eb70 con 0x7ff6280747a0 2024-01-27T04:20:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.199+0000 7ff639ffb640 1 -- 172.21.15.110:0/3266086415 <== osd.0 v2:172.21.15.110:6816/1545415171 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff64413eb70 con 0x7ff6280747a0 2024-01-27T04:20:21.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.199+0000 7ff639ffb640 10 client.17001.objecter ms_dispatch 0x7ff644000b90 command_reply(tid 2: 0 ) v1 2024-01-27T04:20:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.199+0000 7ff639ffb640 10 client.17001.objecter _finish_command 2 = system:0 2024-01-27T04:20:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.199+0000 7ff639ffb640 20 client.17001.objecter put_session s=0x7ff6280740d0 osd=0 4 2024-01-27T04:20:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.199+0000 7ff639ffb640 15 client.17001.objecter _session_command_op_remove 0 2 2024-01-27T04:20:21.201 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-01-27T04:20:21.201 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-27T04:20:21.201 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-01-27T04:20:21.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 10 librados: watch_flush enter 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 10 librados: watch_flush exit 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 10 client.17001.objecter close_session for osd.0 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7ff6280747a0 msgr2=0x7ff628076c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:6816/1545415171,v1:172.21.15.110:6817/1545415171] conn(0x7ff6280747a0 0x7ff628076c40 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 20 client.17001.objecter put_session s=0x7ff6280740d0 osd=0 1 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 20 client.17001.objecter shutdown clearing up homeless session... 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 10 client.17001.objecter successfully canceled tick 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff62806ca20 msgr2=0x7ff62806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff62806ca20 0x7ff62806eee0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff644146700 tx=0x7ff634007d30 comp rx=0 tx=0).stop 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 10 monclient: shutdown 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64406a940 msgr2=0x7ff64413e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 --2- 172.21.15.110:0/3266086415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff64406a940 0x7ff64413e630 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7ff64000c730 tx=0x7ff64000cbf0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff649851640 1 -- 172.21.15.110:0/3266086415 reap_dead start 2024-01-27T04:20:21.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 shutdown_connections 2024-01-27T04:20:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 >> 172.21.15.110:0/3266086415 conn(0x7ff6440b7d70 msgr2=0x7ff6440b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 shutdown_connections 2024-01-27T04:20:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 -- 172.21.15.110:0/3266086415 wait complete. 2024-01-27T04:20:21.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.200+0000 7ff64b2b4640 1 librados: shutdown 2024-01-27T04:20:21.212 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.311+0000 7f2247fa9640 10 monclient: get_monmap_and_config 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.311+0000 7f2247fa9640 10 monclient: build_initial_monmap 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.311+0000 7f2247fa9640 10 monclient: monmap: 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:21.312917+0000 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:21.312917+0000 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:21.313 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:21.314 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:21.314 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.312+0000 7f2247fa9640 1 Processor -- start 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.312+0000 7f2247fa9640 1 -- start start 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.312+0000 7f2247fa9640 10 monclient: init 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22400b4680 0x7f2240142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 10 monclient(hunting): picked mon.noname-b con 0x7f22400b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22400b4a60 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2240142eb0 0x7f22400b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 10 monclient(hunting): picked mon.noname-c con 0x7f2240142eb0 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400afbe0 con 0x7f22400b4a60 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2240142240 con 0x7f2240142eb0 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22400b0890 con 0x7f22400b4680 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2247fa9640 10 monclient(hunting): authenticate will time out at 3704.667132s 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2245d45640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22400b4680 0x7f2240142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.313+0000 7f2246546640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2240142eb0 0x7f22400b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2245d45640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22400b4680 0x7f2240142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40644/0 (socket says 172.21.15.110:40644) 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2245d45640 1 -- 172.21.15.110:0/2352726978 learned_addr learned my addr 172.21.15.110:0/2352726978 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2245d45640 10 monclient(hunting): get_auth_request con 0x7f22400b4680 auth_method 0 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2245d45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2245d45640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2245d45640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2245d45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2245d45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2245d45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2246546640 10 monclient(hunting): get_auth_request con 0x7f2240142eb0 auth_method 0 2024-01-27T04:20:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2246546640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2246546640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.314+0000 7f2246546640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.315+0000 7f2246546640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.315+0000 7f2246546640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.315+0000 7f2246546640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.316+0000 7f2245d45640 10 monclient(hunting): handle_auth_done global_id 18872 payload 274 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.316+0000 7f2245d45640 1 -- 172.21.15.110:0/2352726978 >> v2:172.21.15.162:3300/0 conn(0x7f2240142eb0 msgr2=0x7f22400b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.316+0000 7f2245d45640 1 --2- 172.21.15.110:0/2352726978 >> v2:172.21.15.162:3300/0 conn(0x7f2240142eb0 0x7f22400b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.316+0000 7f2245d45640 1 -- 172.21.15.110:0/2352726978 >> v1:172.21.15.162:6789/0 conn(0x7f22400b4a60 legacy=0x7f22400b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.316+0000 7f2245d45640 10 monclient: _finish_hunting 0 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.316+0000 7f2245d45640 1 monclient: found mon.noname-b 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.316+0000 7f2245d45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.316+0000 7f2245d45640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.316+0000 7f2245d45640 1 -- 172.21.15.110:0/2352726978 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22400b0b00 con 0x7f22400b4680 2024-01-27T04:20:21.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.317+0000 7f2245d45640 1 --2- 172.21.15.110:0/2352726978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22400b4680 0x7f2240142970 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f223c009870 tx=0x7f223c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97675ea7295379c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 1 -- 172.21.15.110:0/2352726978 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c031640 con 0x7f22400b4680 2024-01-27T04:20:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 10 monclient: dump: 2024-01-27T04:20:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 10 monclient: _finish_auth 0 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 10 monclient: _check_auth_tickets 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 1 -- 172.21.15.110:0/2352726978 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223c0317a0 con 0x7f22400b4680 2024-01-27T04:20:21.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2247fa9640 5 monclient: authenticate success, global_id 18872 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2247fa9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2247fa9640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 1 -- 172.21.15.110:0/2352726978 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c031b30 con 0x7f22400b4680 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2247fa9640 4 set_mon_vals no callback set 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.322+0000 7f2244d43640 10 monclient: dump: 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.325 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.326 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.326 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.326 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 10 monclient: shutdown 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 1 -- 172.21.15.110:0/2352726978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22400b4680 msgr2=0x7f2240142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 1 --2- 172.21.15.110:0/2352726978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22400b4680 0x7f2240142970 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f223c009870 tx=0x7f223c02b8f0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 1 -- 172.21.15.110:0/2352726978 shutdown_connections 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 1 --2- 172.21.15.110:0/2352726978 >> v2:172.21.15.162:3300/0 conn(0x7f2240142eb0 0x7f22400b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 1 --2- 172.21.15.110:0/2352726978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22400b4680 0x7f2240142970 unknown :-1 s=CLOSED pgs=3409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 1 -- 172.21.15.110:0/2352726978 >> 172.21.15.110:0/2352726978 conn(0x7f2240066490 msgr2=0x7f22400668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 1 -- 172.21.15.110:0/2352726978 shutdown_connections 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 1 -- 172.21.15.110:0/2352726978 wait complete. 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 10 monclient: build_initial_monmap 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.323+0000 7f2247fa9640 10 monclient: monmap: 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:21.328 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 librados: starting msgr at 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 librados: starting objecter 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 Processor -- start 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 -- start start 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 librados: setting wanted keys 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 librados: calling monclient init 2024-01-27T04:20:21.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient: init 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22400b4680 0x7f224013db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient(hunting): picked mon.noname-a con 0x7f22400b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2240142eb0 0x7f22400ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient(hunting): picked mon.noname-c con 0x7f2240142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22400aa7b0 0x7f22401d7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient(hunting): picked mon.noname-b con 0x7f22400aa7b0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400ab700 con 0x7f22400b4680 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400ab940 con 0x7f22400aa7b0 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22401d7670 con 0x7f2240142eb0 2024-01-27T04:20:21.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:21.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.324+0000 7f2247fa9640 10 monclient(hunting): authenticate will time out at 3704.678870s 2024-01-27T04:20:21.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.328+0000 7f2246546640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22400aa7b0 0x7f22401d7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.328+0000 7f2246546640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22400aa7b0 0x7f22401d7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:58796/0 (socket says 172.21.15.110:58796) 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.328+0000 7f2246546640 1 -- 172.21.15.110:0/2536979173 learned_addr learned my addr 172.21.15.110:0/2536979173 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.328+0000 7f2246546640 10 monclient(hunting): get_auth_request con 0x7f22400aa7b0 auth_method 0 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.328+0000 7f2246546640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.328+0000 7f2246546640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.328+0000 7f2246546640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.329+0000 7f2245544640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2240142eb0 0x7f22400ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.329+0000 7f2245544640 10 monclient(hunting): get_auth_request con 0x7f2240142eb0 auth_method 0 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.329+0000 7f2245544640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.329+0000 7f2245544640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.329+0000 7f2245544640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.329+0000 7f2245544640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.329+0000 7f2245544640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.329+0000 7f2245544640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.364+0000 7f2245d45640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22400b4680 0x7f224013db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.365+0000 7f2245d45640 10 monclient(hunting): get_auth_request con 0x7f22400b4680 auth_method 0 2024-01-27T04:20:21.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.365+0000 7f2245d45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.365+0000 7f2245d45640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.365+0000 7f2245d45640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.365+0000 7f2245d45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.365+0000 7f2245d45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.365+0000 7f2245d45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2246546640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2246546640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2246546640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 10 monclient(hunting): handle_auth_done global_id 17004 payload 824 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 1 -- 172.21.15.110:0/2536979173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2240142eb0 msgr2=0x7f22400ab1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2240142eb0 0x7f22400ab1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 1 -- 172.21.15.110:0/2536979173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22400aa7b0 msgr2=0x7f22401d7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22400aa7b0 0x7f22401d7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 10 monclient: _finish_hunting 0 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 1 monclient: found mon.noname-a 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T04:20:21.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.368+0000 7f2245d45640 1 -- 172.21.15.110:0/2536979173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c0094d0 con 0x7f22400b4680 2024-01-27T04:20:21.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.370+0000 7f2245d45640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22400b4680 0x7f224013db70 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f223c033000 tx=0x7f223c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.370+0000 7f22367fc640 10 client.?.objecter ms_handle_connect 0x7f22400b4680 2024-01-27T04:20:21.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.370+0000 7f22367fc640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 1 -- 172.21.15.110:0/2536979173 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c03ac60 con 0x7f22400b4680 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 10 monclient: dump: 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.376 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 10 monclient: _finish_auth 0 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 10 monclient: _check_auth_tickets 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 1 -- 172.21.15.110:0/2536979173 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223c03ade0 con 0x7f22400b4680 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 1 -- 172.21.15.110:0/2536979173 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c044630 con 0x7f22400b4680 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2247fa9640 5 monclient: authenticate success, global_id 17004 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2247fa9640 10 monclient: _renew_subs 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2247fa9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2237fff640 4 set_mon_vals no callback set 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22401d7950 con 0x7f22400b4680 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f22367fc640 10 monclient: dump: 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2247fa9640 10 client.17004.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2247fa9640 10 monclient: _renew_subs 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2247fa9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22400a94b0 con 0x7f22400b4680 2024-01-27T04:20:21.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.374+0000 7f2247fa9640 1 librados: init done 2024-01-27T04:20:21.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.375+0000 7f2247fa9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:21.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.375+0000 7f2247fa9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:21.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.375+0000 7f2247fa9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:21.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.375+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2240074d10 con 0x7f22400b4680 2024-01-27T04:20:21.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.394+0000 7f22367fc640 1 -- 172.21.15.110:0/2536979173 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f223c02f050 con 0x7f22400b4680 2024-01-27T04:20:21.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.394+0000 7f22367fc640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f222406c9f0 0x7f222406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.394+0000 7f2245544640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f222406c9f0 0x7f222406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.394+0000 7f2245544640 10 monclient: get_auth_request con 0x7f222406c9f0 auth_method 0 2024-01-27T04:20:21.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.395+0000 7f2245544640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f222406c9f0 0x7f222406eeb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f224013efd0 tx=0x7f2230009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.395+0000 7f22367fc640 10 client.17004.objecter ms_handle_connect 0x7f222406c9f0 2024-01-27T04:20:21.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.628+0000 7f22367fc640 1 -- 172.21.15.110:0/2536979173 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f223c07a780 con 0x7f22400b4680 2024-01-27T04:20:21.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.628+0000 7f22367fc640 10 client.17004.objecter ms_dispatch 0x7f2240000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:21.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.628+0000 7f22367fc640 3 client.17004.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:21.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.628+0000 7f22367fc640 3 client.17004.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:21.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.628+0000 7f22367fc640 20 client.17004.objecter dump_active .. 0 homeless 2024-01-27T04:20:21.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.628+0000 7f22367fc640 1 -- 172.21.15.110:0/2536979173 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f223c076a20 con 0x7f22400b4680 2024-01-27T04:20:21.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.628+0000 7f22367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:21.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.628+0000 7f22367fc640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:21.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.803+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f2240069970 con 0x7f222406c9f0 2024-01-27T04:20:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.804+0000 7f22367fc640 1 -- 172.21.15.110:0/2536979173 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2240069970 con 0x7f222406c9f0 2024-01-27T04:20:21.805 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 0 to deep-scrub 2024-01-27T04:20:21.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 10 librados: watch_flush enter 2024-01-27T04:20:21.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 10 librados: watch_flush exit 2024-01-27T04:20:21.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 20 client.17004.objecter shutdown clearing up homeless session... 2024-01-27T04:20:21.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 10 client.17004.objecter successfully canceled tick 2024-01-27T04:20:21.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f222406c9f0 msgr2=0x7f222406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f222406c9f0 0x7f222406eeb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f224013efd0 tx=0x7f2230009040 comp rx=0 tx=0).stop 2024-01-27T04:20:21.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 10 monclient: shutdown 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22400b4680 msgr2=0x7f224013db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22400b4680 0x7f224013db70 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f223c033000 tx=0x7f223c0046c0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 shutdown_connections 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f222406c9f0 0x7f222406eeb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2240142eb0 0x7f22400ab1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22400aa7b0 0x7f22401d7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 --2- 172.21.15.110:0/2536979173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22400b4680 0x7f224013db70 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 >> 172.21.15.110:0/2536979173 conn(0x7f2240066490 msgr2=0x7f224013f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 shutdown_connections 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 -- 172.21.15.110:0/2536979173 wait complete. 2024-01-27T04:20:21.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.806+0000 7f2247fa9640 1 librados: shutdown 2024-01-27T04:20:21.819 INFO:tasks.ceph:Scrubbing osd.1 2024-01-27T04:20:21.819 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-27T04:20:21.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.924+0000 7f63aa1a9640 10 monclient: get_monmap_and_config 2024-01-27T04:20:21.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.924+0000 7f63aa1a9640 10 monclient: build_initial_monmap 2024-01-27T04:20:21.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.924+0000 7f63aa1a9640 10 monclient: monmap: 2024-01-27T04:20:21.925 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:21.925 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.925 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:21.925451+0000 2024-01-27T04:20:21.925 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:21.925451+0000 2024-01-27T04:20:21.926 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:21.926 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.926 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:21.926 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:21.926 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:21.926 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:21.926 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:21.926 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 1 Processor -- start 2024-01-27T04:20:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 1 -- start start 2024-01-27T04:20:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient: init 2024-01-27T04:20:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:21.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a41426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient(hunting): picked mon.noname-b con 0x7f63a406a940 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f63a40a9e40 0x7f63a4142c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient(hunting): picked mon.noname-a con 0x7f63a40a9e40 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f63a40b2d40 0x7f63a40b3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient(hunting): picked mon.noname-c con 0x7f63a40b2d40 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a406ad20 con 0x7f63a40a9e40 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a40aac50 con 0x7f63a40b2d40 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40aaf30 con 0x7f63a406a940 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.925+0000 7f63aa1a9640 10 monclient(hunting): authenticate will time out at 3705.279647s 2024-01-27T04:20:21.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a41426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a3fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f63a40b2d40 0x7f63a40b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a2ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f63a40a9e40 0x7f63a4142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a41426d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40650/0 (socket says 172.21.15.110:40650) 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a37fe640 1 -- 172.21.15.110:0/3093476090 learned_addr learned my addr 172.21.15.110:0/3093476090 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a37fe640 10 monclient(hunting): get_auth_request con 0x7f63a406a940 auth_method 0 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a37fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a2ffd640 10 monclient(hunting): get_auth_request con 0x7f63a40a9e40 auth_method 0 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a3fff640 10 monclient(hunting): get_auth_request con 0x7f63a40b2d40 auth_method 0 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a3fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.926+0000 7f63a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.927+0000 7f63a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.927+0000 7f63a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.927+0000 7f63a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.927+0000 7f63a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.927+0000 7f63a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.927+0000 7f63a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.927+0000 7f63a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.927+0000 7f63a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.927+0000 7f63a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 10 monclient(hunting): handle_auth_done global_id 18878 payload 274 2024-01-27T04:20:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 1 -- 172.21.15.110:0/3093476090 >> v2:172.21.15.162:3300/0 conn(0x7f63a40b2d40 msgr2=0x7f63a40b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 1 --2- 172.21.15.110:0/3093476090 >> v2:172.21.15.162:3300/0 conn(0x7f63a40b2d40 0x7f63a40b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 1 -- 172.21.15.110:0/3093476090 >> v2:172.21.15.110:3300/0 conn(0x7f63a40a9e40 msgr2=0x7f63a4142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 1 --2- 172.21.15.110:0/3093476090 >> v2:172.21.15.110:3300/0 conn(0x7f63a40a9e40 0x7f63a4142c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 10 monclient: _finish_hunting 0 2024-01-27T04:20:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 1 monclient: found mon.noname-b 2024-01-27T04:20:21.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:21.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:21.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a37fe640 1 -- 172.21.15.110:0/3093476090 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a41421b0 con 0x7f63a406a940 2024-01-27T04:20:21.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.928+0000 7f63a3fff640 1 --2- 172.21.15.110:0/3093476090 >> v2:172.21.15.162:3300/0 conn(0x7f63a40b2d40 0x7f63a40b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T04:20:21.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.930+0000 7f63a37fe640 1 --2- 172.21.15.110:0/3093476090 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a41426d0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f63a4146730 tx=0x7f639402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91f3d72477963f4e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 1 -- 172.21.15.110:0/3093476090 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394031870 con 0x7f63a406a940 2024-01-27T04:20:21.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:21.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 10 monclient: dump: 2024-01-27T04:20:21.935 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.935 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.935 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 10 monclient: _finish_auth 0 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 10 monclient: _check_auth_tickets 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 1 -- 172.21.15.110:0/3093476090 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63940319d0 con 0x7f63a406a940 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 1 -- 172.21.15.110:0/3093476090 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394031d60 con 0x7f63a406a940 2024-01-27T04:20:21.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63a27fc640 10 monclient: dump: 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63aa1a9640 5 monclient: authenticate success, global_id 18878 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63aa1a9640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:21.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.934+0000 7f63aa1a9640 4 set_mon_vals no callback set 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 10 monclient: shutdown 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3093476090 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 msgr2=0x7f63a41426d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 1 --2- 172.21.15.110:0/3093476090 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a41426d0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f63a4146730 tx=0x7f639402b910 comp rx=0 tx=0).stop 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3093476090 shutdown_connections 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 1 --2- 172.21.15.110:0/3093476090 >> v2:172.21.15.162:3300/0 conn(0x7f63a40b2d40 0x7f63a40b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 1 --2- 172.21.15.110:0/3093476090 >> v2:172.21.15.110:3300/0 conn(0x7f63a40a9e40 0x7f63a4142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 1 --2- 172.21.15.110:0/3093476090 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a41426d0 unknown :-1 s=CLOSED pgs=3410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3093476090 >> 172.21.15.110:0/3093476090 conn(0x7f63a40b7d70 msgr2=0x7f63a40b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3093476090 shutdown_connections 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3093476090 wait complete. 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 10 monclient: build_initial_monmap 2024-01-27T04:20:21.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.935+0000 7f63aa1a9640 10 monclient: monmap: 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.936+0000 7f63aa1a9640 1 librados: starting msgr at 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.936+0000 7f63aa1a9640 1 librados: starting objecter 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.936+0000 7f63aa1a9640 1 Processor -- start 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.936+0000 7f63aa1a9640 1 -- start start 2024-01-27T04:20:21.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.936+0000 7f63aa1a9640 1 librados: setting wanted keys 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.936+0000 7f63aa1a9640 1 librados: calling monclient init 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.936+0000 7f63aa1a9640 10 monclient: init 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a40affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient(hunting): picked mon.noname-c con 0x7f63a406a940 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63a40a9e40 0x7f63a40b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient(hunting): picked mon.noname-a con 0x7f63a40a9e40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63a40b2d40 0x7f63a41d7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient(hunting): picked mon.noname-b con 0x7f63a40b2d40 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40ad410 con 0x7f63a40a9e40 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40b1170 con 0x7f63a40b2d40 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:21.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a411ffb0 con 0x7f63a406a940 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63aa1a9640 10 monclient(hunting): authenticate will time out at 3705.291210s 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63a2ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63a40a9e40 0x7f63a40b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63a37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a40affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.937+0000 7f63a3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63a40b2d40 0x7f63a41d7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a40affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40660/0 (socket says 172.21.15.110:40660) 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a37fe640 1 -- 172.21.15.110:0/3478532112 learned_addr learned my addr 172.21.15.110:0/3478532112 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a37fe640 10 monclient(hunting): get_auth_request con 0x7f63a406a940 auth_method 0 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a37fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a2ffd640 10 monclient(hunting): get_auth_request con 0x7f63a40a9e40 auth_method 0 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a3fff640 10 monclient(hunting): get_auth_request con 0x7f63a40b2d40 auth_method 0 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a3fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:21.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.938+0000 7f63a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:21.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 10 monclient(hunting): handle_auth_done global_id 20917 payload 824 2024-01-27T04:20:21.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 1 -- 172.21.15.110:0/3478532112 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 msgr2=0x7f63a40affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63a406a940 0x7f63a40affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 1 -- 172.21.15.110:0/3478532112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63a40a9e40 msgr2=0x7f63a40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:21.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63a40a9e40 0x7f63a40b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:21.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 10 monclient: _finish_hunting 0 2024-01-27T04:20:21.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 1 monclient: found mon.noname-b 2024-01-27T04:20:21.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:21.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:21.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.941+0000 7f63a3fff640 1 -- 172.21.15.110:0/3478532112 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c009a70 con 0x7f63a40b2d40 2024-01-27T04:20:21.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.943+0000 7f63a3fff640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63a40b2d40 0x7f63a41d7620 secure :-1 s=READY pgs=5086 cs=0 l=1 rev1=1 crypto rx=0x7f639c00efc0 tx=0x7f639c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.943+0000 7f638bfff640 10 client.?.objecter ms_handle_connect 0x7f63a40b2d40 2024-01-27T04:20:21.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.943+0000 7f638bfff640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:21.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 1 -- 172.21.15.110:0/3478532112 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c00cd30 con 0x7f63a40b2d40 2024-01-27T04:20:21.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 10 monclient: dump: 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 10 monclient: _finish_auth 0 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 10 monclient: _check_auth_tickets 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:21.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f63aa1a9640 5 monclient: authenticate success, global_id 20917 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 1 -- 172.21.15.110:0/3478532112 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c00f040 con 0x7f63a40b2d40 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f63aa1a9640 10 monclient: _renew_subs 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f638bfff640 1 -- 172.21.15.110:0/3478532112 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c003bd0 con 0x7f63a40b2d40 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63940094d0 con 0x7f63a40b2d40 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f638bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f638bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f638bfff640 10 monclient: dump: 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:21.950 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.946+0000 7f63a1ffb640 4 set_mon_vals no callback set 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 10 client.20917.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 10 monclient: _renew_subs 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a41d7b60 con 0x7f63a40b2d40 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 1 librados: init done 2024-01-27T04:20:21.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 10 client.20917.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:21.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 15 client.20917.objecter _session_command_op_assign -1 1 2024-01-27T04:20:21.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 20 client.20917.objecter _get_session osd=-1 returning homeless 2024-01-27T04:20:21.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 10 client.20917.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:21.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:21.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.947+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63a40b2770 con 0x7f63a40b2d40 2024-01-27T04:20:21.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.966+0000 7f638bfff640 1 -- 172.21.15.110:0/3478532112 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f639c014480 con 0x7f63a40b2d40 2024-01-27T04:20:21.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.966+0000 7f638bfff640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639006c9f0 0x7f639006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.966+0000 7f63a37fe640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639006c9f0 0x7f639006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.966+0000 7f63a37fe640 10 monclient: get_auth_request con 0x7f639006c9f0 auth_method 0 2024-01-27T04:20:21.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.967+0000 7f63a37fe640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639006c9f0 0x7f639006eeb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f639402b950 tx=0x7f639403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.967+0000 7f638bfff640 10 client.20917.objecter ms_handle_connect 0x7f639006c9f0 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 1 -- 172.21.15.110:0/3478532112 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f639c059850 con 0x7f63a40b2d40 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 10 client.20917.objecter ms_dispatch 0x7f63a4000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 3 client.20917.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 3 client.20917.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 10 client.20917.objecter checking command 1 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6390074770 0x7f6390076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 20 client.20917.objecter _get_session s=0x7f63900740a0 osd=1 3 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 20 client.20917.objecter put_session s=0x7f63900740a0 osd=1 3 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 15 client.20917.objecter _session_command_op_remove -1 1 2024-01-27T04:20:21.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 20 client.20917.objecter _get_session s=0x7f63900740a0 osd=1 3 2024-01-27T04:20:21.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 20 client.20917.objecter get_session s=0x7f63900740a0 osd=1 3 2024-01-27T04:20:21.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 15 client.20917.objecter _session_command_op_assign 1 1 2024-01-27T04:20:21.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 20 client.20917.objecter put_session s=0x7f63900740a0 osd=1 4 2024-01-27T04:20:21.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 10 client.20917.objecter _send_command 1 2024-01-27T04:20:21.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 1 -- 172.21.15.110:0/3478532112 --> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63900772e0 con 0x7f6390074770 2024-01-27T04:20:21.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f638bfff640 20 client.20917.objecter dump_active .. 0 homeless 2024-01-27T04:20:21.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.968+0000 7f63a2ffd640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6390074770 0x7f6390076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:21.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.969+0000 7f63a2ffd640 10 monclient: get_auth_request con 0x7f6390074770 auth_method 0 2024-01-27T04:20:21.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.970+0000 7f638bfff640 1 -- 172.21.15.110:0/3478532112 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=1334) v2 ==== 24+0+0 (secure 0 0 0) 0x7f639c055af0 con 0x7f63a40b2d40 2024-01-27T04:20:21.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.970+0000 7f638bfff640 10 monclient: handle_get_version_reply finishing 1 version 1334 2024-01-27T04:20:21.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.971+0000 7f63a2ffd640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6390074770 0x7f6390076c10 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:21.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.971+0000 7f638bfff640 10 client.20917.objecter ms_handle_connect 0x7f6390074770 2024-01-27T04:20:21.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.994+0000 7f638bfff640 1 -- 172.21.15.110:0/3478532112 <== osd.1 v2:172.21.15.110:6823/518896551 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f63900772e0 con 0x7f6390074770 2024-01-27T04:20:21.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.994+0000 7f638bfff640 10 client.20917.objecter ms_dispatch 0x7f63a4000b90 command_reply(tid 1: 0 ) v1 2024-01-27T04:20:21.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.994+0000 7f638bfff640 10 client.20917.objecter _finish_command 1 = system:0 2024-01-27T04:20:21.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.994+0000 7f638bfff640 20 client.20917.objecter put_session s=0x7f63900740a0 osd=1 4 2024-01-27T04:20:21.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:21.994+0000 7f638bfff640 15 client.20917.objecter _session_command_op_remove 1 1 2024-01-27T04:20:22.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 10 client.20917.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-27T04:20:22.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 15 client.20917.objecter _session_command_op_assign -1 2 2024-01-27T04:20:22.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 20 client.20917.objecter _get_session s=0x7f63900740a0 osd=1 3 2024-01-27T04:20:22.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 20 client.20917.objecter put_session s=0x7f63900740a0 osd=1 3 2024-01-27T04:20:22.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 20 client.20917.objecter _get_session s=0x7f63900740a0 osd=1 3 2024-01-27T04:20:22.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 15 client.20917.objecter _session_command_op_remove -1 2 2024-01-27T04:20:22.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 20 client.20917.objecter get_session s=0x7f63900740a0 osd=1 3 2024-01-27T04:20:22.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 15 client.20917.objecter _session_command_op_assign 1 2 2024-01-27T04:20:22.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 20 client.20917.objecter put_session s=0x7f63900740a0 osd=1 4 2024-01-27T04:20:22.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 10 client.20917.objecter _send_command 2 2024-01-27T04:20:22.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.007+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 --> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f63a4074d10 con 0x7f6390074770 2024-01-27T04:20:22.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.022+0000 7f638bfff640 1 -- 172.21.15.110:0/3478532112 <== osd.1 v2:172.21.15.110:6823/518896551 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f63a4074d10 con 0x7f6390074770 2024-01-27T04:20:22.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.022+0000 7f638bfff640 10 client.20917.objecter ms_dispatch 0x7f63a4000b90 command_reply(tid 2: 0 ) v1 2024-01-27T04:20:22.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.022+0000 7f638bfff640 10 client.20917.objecter _finish_command 2 = system:0 2024-01-27T04:20:22.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.022+0000 7f638bfff640 20 client.20917.objecter put_session s=0x7f63900740a0 osd=1 4 2024-01-27T04:20:22.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.022+0000 7f638bfff640 15 client.20917.objecter _session_command_op_remove 1 2 2024-01-27T04:20:22.024 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-01-27T04:20:22.024 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-27T04:20:22.024 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-01-27T04:20:22.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 10 librados: watch_flush enter 2024-01-27T04:20:22.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 10 librados: watch_flush exit 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 10 client.20917.objecter close_session for osd.1 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6390074770 msgr2=0x7f6390076c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.110:6823/518896551,v1:172.21.15.110:6825/518896551] conn(0x7f6390074770 0x7f6390076c10 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 20 client.20917.objecter put_session s=0x7f63900740a0 osd=1 1 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 20 client.20917.objecter shutdown clearing up homeless session... 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 10 client.20917.objecter successfully canceled tick 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639006c9f0 msgr2=0x7f639006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639006c9f0 0x7f639006eeb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f639402b950 tx=0x7f639403c040 comp rx=0 tx=0).stop 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 10 monclient: shutdown 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63a40b2d40 msgr2=0x7f63a41d7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 --2- 172.21.15.110:0/3478532112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63a40b2d40 0x7f63a41d7620 secure :-1 s=READY pgs=5086 cs=0 l=1 rev1=1 crypto rx=0x7f639c00efc0 tx=0x7f639c00c490 comp rx=0 tx=0).stop 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63a3fff640 1 -- 172.21.15.110:0/3478532112 reap_dead start 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 shutdown_connections 2024-01-27T04:20:22.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 >> 172.21.15.110:0/3478532112 conn(0x7f63a40b7d70 msgr2=0x7f63a40b8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:22.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 shutdown_connections 2024-01-27T04:20:22.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 -- 172.21.15.110:0/3478532112 wait complete. 2024-01-27T04:20:22.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.023+0000 7f63aa1a9640 1 librados: shutdown 2024-01-27T04:20:22.036 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-27T04:20:22.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.137+0000 7fb22834e640 10 monclient: get_monmap_and_config 2024-01-27T04:20:22.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.137+0000 7fb22834e640 10 monclient: build_initial_monmap 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.137+0000 7fb22834e640 10 monclient: monmap: 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:22.138676+0000 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:22.138676+0000 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:22.139 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 1 Processor -- start 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 1 -- start start 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient: init 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb22006a6d0 0x7fb2200b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient(hunting): picked mon.noname-c con 0x7fb22006a6d0 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006aab0 0x7fb220142250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient(hunting): picked mon.noname-b con 0x7fb22006aab0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2200b4000 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2200aaea0 con 0x7fb2200b4000 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb220142ea0 con 0x7fb22006a6d0 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2200ab0f0 con 0x7fb22006aab0 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.138+0000 7fb22834e640 10 monclient(hunting): authenticate will time out at 3705.492827s 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.139+0000 7fb2258e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006aab0 0x7fb220142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.139+0000 7fb2260ea640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb22006a6d0 0x7fb2200b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.139+0000 7fb2258e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006aab0 0x7fb220142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40676/0 (socket says 172.21.15.110:40676) 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.139+0000 7fb2258e9640 1 -- 172.21.15.110:0/1614876188 learned_addr learned my addr 172.21.15.110:0/1614876188 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.139+0000 7fb2258e9640 10 monclient(hunting): get_auth_request con 0x7fb22006aab0 auth_method 0 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.139+0000 7fb2258e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.139+0000 7fb2258e9640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.139+0000 7fb2258e9640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.140+0000 7fb2258e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.140+0000 7fb2258e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.140+0000 7fb2258e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.140+0000 7fb2260ea640 10 monclient(hunting): get_auth_request con 0x7fb22006a6d0 auth_method 0 2024-01-27T04:20:22.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.140+0000 7fb2260ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.140+0000 7fb2260ea640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.140+0000 7fb2260ea640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.141+0000 7fb2260ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.141+0000 7fb2260ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.141+0000 7fb2260ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.141+0000 7fb2258e9640 10 monclient(hunting): handle_auth_done global_id 18884 payload 274 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.141+0000 7fb2258e9640 1 -- 172.21.15.110:0/1614876188 >> v2:172.21.15.162:3300/0 conn(0x7fb22006a6d0 msgr2=0x7fb2200b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.141+0000 7fb2258e9640 1 --2- 172.21.15.110:0/1614876188 >> v2:172.21.15.162:3300/0 conn(0x7fb22006a6d0 0x7fb2200b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.141+0000 7fb2258e9640 1 -- 172.21.15.110:0/1614876188 >> v1:172.21.15.162:6789/0 conn(0x7fb2200b4000 legacy=0x7fb220142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.142+0000 7fb2258e9640 10 monclient: _finish_hunting 0 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.142+0000 7fb2258e9640 1 monclient: found mon.noname-b 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.142+0000 7fb2258e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.142+0000 7fb2258e9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.142+0000 7fb2258e9640 1 -- 172.21.15.110:0/1614876188 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2200b44c0 con 0x7fb22006aab0 2024-01-27T04:20:22.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.143+0000 7fb2258e9640 1 --2- 172.21.15.110:0/1614876188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006aab0 0x7fb220142250 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fb210009870 tx=0x7fb21002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=685b1b37c944f7d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:22.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 1 -- 172.21.15.110:0/1614876188 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100317c0 con 0x7fb22006aab0 2024-01-27T04:20:22.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:22.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 10 monclient: dump: 2024-01-27T04:20:22.148 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.148 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.148 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 10 monclient: _finish_auth 0 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 10 monclient: _check_auth_tickets 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 1 -- 172.21.15.110:0/1614876188 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb210031920 con 0x7fb22006aab0 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb22834e640 5 monclient: authenticate success, global_id 18884 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb22834e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb22834e640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:22.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 1 -- 172.21.15.110:0/1614876188 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210031cb0 con 0x7fb22006aab0 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb22834e640 4 set_mon_vals no callback set 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb2250e8640 10 monclient: dump: 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.150 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.151 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb22834e640 10 monclient: shutdown 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb22834e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb22834e640 1 -- 172.21.15.110:0/1614876188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006aab0 msgr2=0x7fb220142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.147+0000 7fb22834e640 1 --2- 172.21.15.110:0/1614876188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006aab0 0x7fb220142250 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fb210009870 tx=0x7fb21002b860 comp rx=0 tx=0).stop 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.148+0000 7fb22834e640 1 -- 172.21.15.110:0/1614876188 shutdown_connections 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.148+0000 7fb22834e640 1 --2- 172.21.15.110:0/1614876188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006aab0 0x7fb220142250 unknown :-1 s=CLOSED pgs=3411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.148+0000 7fb22834e640 1 --2- 172.21.15.110:0/1614876188 >> v2:172.21.15.162:3300/0 conn(0x7fb22006a6d0 0x7fb2200b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.148+0000 7fb22834e640 1 -- 172.21.15.110:0/1614876188 >> 172.21.15.110:0/1614876188 conn(0x7fb2200b7e70 msgr2=0x7fb2200b8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.148+0000 7fb22834e640 1 -- 172.21.15.110:0/1614876188 shutdown_connections 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.148+0000 7fb22834e640 1 -- 172.21.15.110:0/1614876188 wait complete. 2024-01-27T04:20:22.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.148+0000 7fb22834e640 10 monclient: build_initial_monmap 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.148+0000 7fb22834e640 10 monclient: monmap: 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 librados: starting msgr at 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 librados: starting objecter 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 Processor -- start 2024-01-27T04:20:22.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 -- start start 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 librados: setting wanted keys 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 librados: calling monclient init 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient: init 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006a6d0 0x7fb220145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient(hunting): picked mon.noname-c con 0x7fb22006a6d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2200b4000 0x7fb22013b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient(hunting): picked mon.noname-b con 0x7fb2200b4000 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2201445f0 0x7fb22012fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient(hunting): picked mon.noname-a con 0x7fb2201445f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22013ca70 con 0x7fb2201445f0 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2201456e0 con 0x7fb2200b4000 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb220130200 con 0x7fb22006a6d0 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.149+0000 7fb22834e640 10 monclient(hunting): authenticate will time out at 3705.503739s 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2260ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006a6d0 0x7fb220145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2258e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2200b4000 0x7fb22013b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2268eb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2201445f0 0x7fb22012fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2258e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2200b4000 0x7fb22013b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:58842/0 (socket says 172.21.15.110:58842) 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2258e9640 1 -- 172.21.15.110:0/2205827580 learned_addr learned my addr 172.21.15.110:0/2205827580 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2258e9640 10 monclient(hunting): get_auth_request con 0x7fb2200b4000 auth_method 0 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2258e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2258e9640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2258e9640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2260ea640 10 monclient(hunting): get_auth_request con 0x7fb22006a6d0 auth_method 0 2024-01-27T04:20:22.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2260ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2260ea640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2260ea640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2268eb640 10 monclient(hunting): get_auth_request con 0x7fb2201445f0 auth_method 0 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2268eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2268eb640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.150+0000 7fb2268eb640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.151+0000 7fb2258e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.151+0000 7fb2258e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.151+0000 7fb2258e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.151+0000 7fb2260ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.151+0000 7fb2260ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.151+0000 7fb2260ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.151+0000 7fb2268eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.151+0000 7fb2268eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.151+0000 7fb2268eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 10 monclient(hunting): handle_auth_done global_id 20926 payload 824 2024-01-27T04:20:22.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 1 -- 172.21.15.110:0/2205827580 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006a6d0 msgr2=0x7fb220145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006a6d0 0x7fb220145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 1 -- 172.21.15.110:0/2205827580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2201445f0 msgr2=0x7fb22012fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2201445f0 0x7fb22012fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 10 monclient: _finish_hunting 0 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 1 monclient: found mon.noname-b 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.153+0000 7fb2258e9640 1 -- 172.21.15.110:0/2205827580 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2100094d0 con 0x7fb2200b4000 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.155+0000 7fb2258e9640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2200b4000 0x7fb22013b440 secure :-1 s=READY pgs=5088 cs=0 l=1 rev1=1 crypto rx=0x7fb21002b8a0 tx=0x7fb2100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.155+0000 7fb216ffd640 10 client.?.objecter ms_handle_connect 0x7fb2200b4000 2024-01-27T04:20:22.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.155+0000 7fb216ffd640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:22.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 1 -- 172.21.15.110:0/2205827580 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210032860 con 0x7fb2200b4000 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 10 monclient: dump: 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 10 monclient: _finish_auth 0 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 10 monclient: _check_auth_tickets 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 1 -- 172.21.15.110:0/2205827580 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2100329c0 con 0x7fb2200b4000 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb22834e640 5 monclient: authenticate success, global_id 20926 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb22834e640 10 monclient: _renew_subs 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb22834e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2201303c0 con 0x7fb2200b4000 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 1 -- 172.21.15.110:0/2205827580 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210032d70 con 0x7fb2200b4000 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb216ffd640 10 monclient: dump: 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.163 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb2248e7640 4 set_mon_vals no callback set 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb22834e640 10 client.20926.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb22834e640 10 monclient: _renew_subs 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb22834e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2201dae70 con 0x7fb2200b4000 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.159+0000 7fb22834e640 1 librados: init done 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.160+0000 7fb22834e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:22.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.160+0000 7fb22834e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:22.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.160+0000 7fb22834e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.160+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22004ea50 con 0x7fb2200b4000 2024-01-27T04:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.406+0000 7fb216ffd640 1 -- 172.21.15.110:0/2205827580 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fb21004e450 con 0x7fb2200b4000 2024-01-27T04:20:22.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.406+0000 7fb216ffd640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb20406ca20 0x7fb20406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.406+0000 7fb2260ea640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb20406ca20 0x7fb20406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.406+0000 7fb2260ea640 10 monclient: get_auth_request con 0x7fb20406ca20 auth_method 0 2024-01-27T04:20:22.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.407+0000 7fb216ffd640 10 client.20926.objecter ms_handle_connect 0x7fb20406ca20 2024-01-27T04:20:22.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.407+0000 7fb2260ea640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb20406ca20 0x7fb20406eee0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009aa0 tx=0x7fb21c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.408+0000 7fb216ffd640 1 -- 172.21.15.110:0/2205827580 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7fb21007a740 con 0x7fb2200b4000 2024-01-27T04:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.408+0000 7fb216ffd640 10 client.20926.objecter ms_dispatch 0x7fb220000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:22.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.408+0000 7fb216ffd640 3 client.20926.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:22.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.408+0000 7fb216ffd640 3 client.20926.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:22.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.408+0000 7fb216ffd640 20 client.20926.objecter dump_active .. 0 homeless 2024-01-27T04:20:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.418+0000 7fb216ffd640 1 -- 172.21.15.110:0/2205827580 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fb2100769e0 con 0x7fb2200b4000 2024-01-27T04:20:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.418+0000 7fb216ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:22.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.418+0000 7fb216ffd640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:22.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.620+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2200b2410 con 0x7fb20406ca20 2024-01-27T04:20:22.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.621+0000 7fb216ffd640 1 -- 172.21.15.110:0/2205827580 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb21c011a00 con 0x7fb20406ca20 2024-01-27T04:20:22.622 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 1 to deep-scrub 2024-01-27T04:20:22.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 10 librados: watch_flush enter 2024-01-27T04:20:22.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 10 librados: watch_flush exit 2024-01-27T04:20:22.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 20 client.20926.objecter shutdown clearing up homeless session... 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 10 client.20926.objecter successfully canceled tick 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb20406ca20 msgr2=0x7fb20406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb20406ca20 0x7fb20406eee0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009aa0 tx=0x7fb21c009190 comp rx=0 tx=0).stop 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 10 monclient: shutdown 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2200b4000 msgr2=0x7fb22013b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2200b4000 0x7fb22013b440 secure :-1 s=READY pgs=5088 cs=0 l=1 rev1=1 crypto rx=0x7fb21002b8a0 tx=0x7fb2100046c0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 shutdown_connections 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb20406ca20 0x7fb20406eee0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2201445f0 0x7fb22012fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2200b4000 0x7fb22013b440 unknown :-1 s=CLOSED pgs=5088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 --2- 172.21.15.110:0/2205827580 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb22006a6d0 0x7fb220145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.623+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 >> 172.21.15.110:0/2205827580 conn(0x7fb2200b7e70 msgr2=0x7fb220133ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:22.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.624+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 shutdown_connections 2024-01-27T04:20:22.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.624+0000 7fb22834e640 1 -- 172.21.15.110:0/2205827580 wait complete. 2024-01-27T04:20:22.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.624+0000 7fb22834e640 1 librados: shutdown 2024-01-27T04:20:22.638 INFO:tasks.ceph:Scrubbing osd.2 2024-01-27T04:20:22.638 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-27T04:20:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.739+0000 7feb94733640 10 monclient: get_monmap_and_config 2024-01-27T04:20:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.739+0000 7feb94733640 10 monclient: build_initial_monmap 2024-01-27T04:20:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.739+0000 7feb94733640 10 monclient: monmap: 2024-01-27T04:20:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:22.740311+0000 2024-01-27T04:20:22.740 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:22.740311+0000 2024-01-27T04:20:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:22.741 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:22.741 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.739+0000 7feb94733640 1 Processor -- start 2024-01-27T04:20:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.739+0000 7feb94733640 1 -- start start 2024-01-27T04:20:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.739+0000 7feb94733640 10 monclient: init 2024-01-27T04:20:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-01-27T04:20:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb8c06a940 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 10 monclient(hunting): picked mon.noname-b con 0x7feb8c142de0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7feb8c0a9e40 0x7feb8c0b3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 10 monclient(hunting): picked mon.noname-c con 0x7feb8c0a9e40 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c0b37c0 con 0x7feb8c06a940 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8c06ad20 con 0x7feb8c0a9e40 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0aac50 con 0x7feb8c142de0 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.740+0000 7feb94733640 10 monclient(hunting): authenticate will time out at 3706.094532s 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb91cce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb92cd0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7feb8c0a9e40 0x7feb8c0b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb91cce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c0b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40694/0 (socket says 172.21.15.110:40694) 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb91cce640 1 -- 172.21.15.110:0/146418416 learned_addr learned my addr 172.21.15.110:0/146418416 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb91cce640 10 monclient(hunting): get_auth_request con 0x7feb8c142de0 auth_method 0 2024-01-27T04:20:22.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb91cce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb91cce640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb91cce640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb92cd0640 10 monclient(hunting): get_auth_request con 0x7feb8c0a9e40 auth_method 0 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb92cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb92cd0640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.741+0000 7feb92cd0640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.742+0000 7feb91cce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.742+0000 7feb91cce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.742+0000 7feb91cce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.742+0000 7feb92cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.742+0000 7feb92cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.742+0000 7feb92cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.743+0000 7feb91cce640 10 monclient(hunting): handle_auth_done global_id 18890 payload 274 2024-01-27T04:20:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.743+0000 7feb91cce640 1 -- 172.21.15.110:0/146418416 >> v2:172.21.15.162:3300/0 conn(0x7feb8c0a9e40 msgr2=0x7feb8c0b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.743+0000 7feb91cce640 1 --2- 172.21.15.110:0/146418416 >> v2:172.21.15.162:3300/0 conn(0x7feb8c0a9e40 0x7feb8c0b3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.743+0000 7feb91cce640 1 -- 172.21.15.110:0/146418416 >> v1:172.21.15.162:6789/0 conn(0x7feb8c06a940 legacy=0x7feb8c1426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.743+0000 7feb91cce640 10 monclient: _finish_hunting 0 2024-01-27T04:20:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.743+0000 7feb91cce640 1 monclient: found mon.noname-b 2024-01-27T04:20:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.743+0000 7feb91cce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.743+0000 7feb91cce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:22.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.743+0000 7feb91cce640 1 -- 172.21.15.110:0/146418416 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0aaf30 con 0x7feb8c142de0 2024-01-27T04:20:22.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.745+0000 7feb91cce640 1 --2- 172.21.15.110:0/146418416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c0b2d40 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009870 tx=0x7feb7c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61f226d239de5b6a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 1 -- 172.21.15.110:0/146418416 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c030840 con 0x7feb8c142de0 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 10 monclient: dump: 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.750 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 10 monclient: _finish_auth 0 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 10 monclient: _check_auth_tickets 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 1 -- 172.21.15.110:0/146418416 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c0309a0 con 0x7feb8c142de0 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 5 monclient: authenticate success, global_id 18890 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 1 -- 172.21.15.110:0/146418416 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c030d30 con 0x7feb8c142de0 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 4 set_mon_vals no callback set 2024-01-27T04:20:22.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb914cd640 10 monclient: dump: 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.752 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 10 monclient: shutdown 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 1 -- 172.21.15.110:0/146418416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 msgr2=0x7feb8c0b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 1 --2- 172.21.15.110:0/146418416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c0b2d40 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009870 tx=0x7feb7c02b980 comp rx=0 tx=0).stop 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 1 -- 172.21.15.110:0/146418416 shutdown_connections 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 1 --2- 172.21.15.110:0/146418416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c0b2d40 unknown :-1 s=CLOSED pgs=3412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 1 --2- 172.21.15.110:0/146418416 >> v2:172.21.15.162:3300/0 conn(0x7feb8c0a9e40 0x7feb8c0b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 1 -- 172.21.15.110:0/146418416 >> 172.21.15.110:0/146418416 conn(0x7feb8c0b7d70 msgr2=0x7feb8c0b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 1 -- 172.21.15.110:0/146418416 shutdown_connections 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.749+0000 7feb94733640 1 -- 172.21.15.110:0/146418416 wait complete. 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.750+0000 7feb94733640 10 monclient: build_initial_monmap 2024-01-27T04:20:22.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.750+0000 7feb94733640 10 monclient: monmap: 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.750+0000 7feb94733640 1 librados: starting msgr at 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.750+0000 7feb94733640 1 librados: starting objecter 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.750+0000 7feb94733640 1 Processor -- start 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.750+0000 7feb94733640 1 -- start start 2024-01-27T04:20:22.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.750+0000 7feb94733640 1 librados: setting wanted keys 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.750+0000 7feb94733640 1 librados: calling monclient init 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.750+0000 7feb94733640 10 monclient: init 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb8c06a940 0x7feb8c0ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient(hunting): picked mon.noname-b con 0x7feb8c06a940 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb8c0a9e40 0x7feb8c1dadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient(hunting): picked mon.noname-a con 0x7feb8c0a9e40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c1d7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient(hunting): picked mon.noname-c con 0x7feb8c142de0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c1da6a0 con 0x7feb8c0a9e40 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c1da8b0 con 0x7feb8c06a940 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c13efa0 con 0x7feb8c142de0 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb94733640 10 monclient(hunting): authenticate will time out at 3706.105454s 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.751+0000 7feb92cd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c1d7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb924cf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb8c06a940 0x7feb8c0ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb91cce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb8c0a9e40 0x7feb8c1dadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb92cd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c1d7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40706/0 (socket says 172.21.15.110:40706) 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb92cd0640 1 -- 172.21.15.110:0/3428497400 learned_addr learned my addr 172.21.15.110:0/3428497400 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb92cd0640 10 monclient(hunting): get_auth_request con 0x7feb8c142de0 auth_method 0 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb92cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb92cd0640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb92cd0640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb924cf640 10 monclient(hunting): get_auth_request con 0x7feb8c06a940 auth_method 0 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb924cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb924cf640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb924cf640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb91cce640 10 monclient(hunting): get_auth_request con 0x7feb8c0a9e40 auth_method 0 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb91cce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb91cce640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb91cce640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb92cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb92cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb92cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb924cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb924cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb924cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb91cce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.752+0000 7feb91cce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.753+0000 7feb91cce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 10 monclient(hunting): handle_auth_done global_id 20932 payload 824 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 1 -- 172.21.15.110:0/3428497400 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 msgr2=0x7feb8c1d7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb8c142de0 0x7feb8c1d7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 1 -- 172.21.15.110:0/3428497400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb8c0a9e40 msgr2=0x7feb8c1dadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb8c0a9e40 0x7feb8c1dadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 10 monclient: _finish_hunting 0 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 1 monclient: found mon.noname-b 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.755+0000 7feb924cf640 1 -- 172.21.15.110:0/3428497400 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c0094d0 con 0x7feb8c06a940 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.757+0000 7feb924cf640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb8c06a940 0x7feb8c0ab900 secure :-1 s=READY pgs=5090 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c3c0 tx=0x7feb8800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.757+0000 7feb82ffd640 10 client.?.objecter ms_handle_connect 0x7feb8c06a940 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.757+0000 7feb82ffd640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 1 -- 172.21.15.110:0/3428497400 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb880052d0 con 0x7feb8c06a940 2024-01-27T04:20:22.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 10 monclient: dump: 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 10 monclient: _finish_auth 0 2024-01-27T04:20:22.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 10 monclient: _check_auth_tickets 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 1 -- 172.21.15.110:0/3428497400 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb88005450 con 0x7feb8c06a940 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 5 monclient: authenticate success, global_id 20932 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 10 monclient: _renew_subs 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c1db310 con 0x7feb8c06a940 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 1 -- 172.21.15.110:0/3428497400 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb88007260 con 0x7feb8c06a940 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb82ffd640 10 monclient: dump: 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.765 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb90ccc640 4 set_mon_vals no callback set 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 10 client.20932.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 10 monclient: _renew_subs 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c13c2c0 con 0x7feb8c06a940 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 1 librados: init done 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 10 client.20932.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 15 client.20932.objecter _session_command_op_assign -1 1 2024-01-27T04:20:22.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 20 client.20932.objecter _get_session osd=-1 returning homeless 2024-01-27T04:20:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 10 client.20932.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.761+0000 7feb94733640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.762+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb8c0b4590 con 0x7feb8c06a940 2024-01-27T04:20:22.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.780+0000 7feb82ffd640 1 -- 172.21.15.110:0/3428497400 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7feb880073c0 con 0x7feb8c06a940 2024-01-27T04:20:22.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.780+0000 7feb82ffd640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7feb7006c9f0 0x7feb7006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.781+0000 7feb91cce640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7feb7006c9f0 0x7feb7006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.781+0000 7feb91cce640 10 monclient: get_auth_request con 0x7feb7006c9f0 auth_method 0 2024-01-27T04:20:22.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.781+0000 7feb91cce640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7feb7006c9f0 0x7feb7006eeb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7feb7c036000 tx=0x7feb7c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:22.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.781+0000 7feb82ffd640 10 client.20932.objecter ms_handle_connect 0x7feb7006c9f0 2024-01-27T04:20:22.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 1 -- 172.21.15.110:0/3428497400 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7feb88058110 con 0x7feb8c06a940 2024-01-27T04:20:22.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 10 client.20932.objecter ms_dispatch 0x7feb8c000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:22.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 3 client.20932.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:22.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 3 client.20932.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:22.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 10 client.20932.objecter checking command 1 2024-01-27T04:20:22.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7feb700746e0 0x7feb70076b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 20 client.20932.objecter _get_session s=0x7feb70074010 osd=2 3 2024-01-27T04:20:22.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 20 client.20932.objecter put_session s=0x7feb70074010 osd=2 3 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 15 client.20932.objecter _session_command_op_remove -1 1 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 20 client.20932.objecter _get_session s=0x7feb70074010 osd=2 3 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 20 client.20932.objecter get_session s=0x7feb70074010 osd=2 3 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 15 client.20932.objecter _session_command_op_assign 2 1 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 20 client.20932.objecter put_session s=0x7feb70074010 osd=2 4 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 10 client.20932.objecter _send_command 1 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 1 -- 172.21.15.110:0/3428497400 --> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb70077250 con 0x7feb700746e0 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.782+0000 7feb82ffd640 20 client.20932.objecter dump_active .. 0 homeless 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.783+0000 7feb92cd0640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7feb700746e0 0x7feb70076b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.783+0000 7feb92cd0640 10 monclient: get_auth_request con 0x7feb700746e0 auth_method 0 2024-01-27T04:20:22.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.785+0000 7feb82ffd640 1 -- 172.21.15.110:0/3428497400 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=1334) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb88001000 con 0x7feb8c06a940 2024-01-27T04:20:22.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.786+0000 7feb82ffd640 10 monclient: handle_get_version_reply finishing 1 version 1334 2024-01-27T04:20:22.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.786+0000 7feb82ffd640 10 client.20932.objecter ms_handle_connect 0x7feb700746e0 2024-01-27T04:20:22.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.786+0000 7feb92cd0640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7feb700746e0 0x7feb70076b80 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:22.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.811+0000 7feb82ffd640 1 -- 172.21.15.110:0/3428497400 <== osd.2 v2:172.21.15.110:6808/3683566991 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7feb70077250 con 0x7feb700746e0 2024-01-27T04:20:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.811+0000 7feb82ffd640 10 client.20932.objecter ms_dispatch 0x7feb8c000b90 command_reply(tid 1: 0 ) v1 2024-01-27T04:20:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.811+0000 7feb82ffd640 10 client.20932.objecter _finish_command 1 = system:0 2024-01-27T04:20:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.811+0000 7feb82ffd640 20 client.20932.objecter put_session s=0x7feb70074010 osd=2 4 2024-01-27T04:20:22.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.811+0000 7feb82ffd640 15 client.20932.objecter _session_command_op_remove 2 1 2024-01-27T04:20:22.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 10 client.20932.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-27T04:20:22.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 15 client.20932.objecter _session_command_op_assign -1 2 2024-01-27T04:20:22.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 20 client.20932.objecter _get_session s=0x7feb70074010 osd=2 3 2024-01-27T04:20:22.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 20 client.20932.objecter put_session s=0x7feb70074010 osd=2 3 2024-01-27T04:20:22.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 20 client.20932.objecter _get_session s=0x7feb70074010 osd=2 3 2024-01-27T04:20:22.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 15 client.20932.objecter _session_command_op_remove -1 2 2024-01-27T04:20:22.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 20 client.20932.objecter get_session s=0x7feb70074010 osd=2 3 2024-01-27T04:20:22.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 15 client.20932.objecter _session_command_op_assign 2 2 2024-01-27T04:20:22.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 20 client.20932.objecter put_session s=0x7feb70074010 osd=2 4 2024-01-27T04:20:22.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 10 client.20932.objecter _send_command 2 2024-01-27T04:20:22.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.833+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 --> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7feb8c1db610 con 0x7feb700746e0 2024-01-27T04:20:22.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb82ffd640 1 -- 172.21.15.110:0/3428497400 <== osd.2 v2:172.21.15.110:6808/3683566991 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7feb8c1db610 con 0x7feb700746e0 2024-01-27T04:20:22.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb82ffd640 10 client.20932.objecter ms_dispatch 0x7feb8c000b90 command_reply(tid 2: 0 ) v1 2024-01-27T04:20:22.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb82ffd640 10 client.20932.objecter _finish_command 2 = system:0 2024-01-27T04:20:22.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb82ffd640 20 client.20932.objecter put_session s=0x7feb70074010 osd=2 4 2024-01-27T04:20:22.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb82ffd640 15 client.20932.objecter _session_command_op_remove 2 2 2024-01-27T04:20:22.851 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-01-27T04:20:22.852 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-27T04:20:22.852 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-01-27T04:20:22.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 10 librados: watch_flush enter 2024-01-27T04:20:22.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 10 librados: watch_flush exit 2024-01-27T04:20:22.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 10 client.20932.objecter close_session for osd.2 2024-01-27T04:20:22.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7feb700746e0 msgr2=0x7feb70076b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.110:6808/3683566991,v1:172.21.15.110:6809/3683566991] conn(0x7feb700746e0 0x7feb70076b80 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 20 client.20932.objecter put_session s=0x7feb70074010 osd=2 1 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 20 client.20932.objecter shutdown clearing up homeless session... 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 10 client.20932.objecter successfully canceled tick 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7feb7006c9f0 msgr2=0x7feb7006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7feb7006c9f0 0x7feb7006eeb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7feb7c036000 tx=0x7feb7c008030 comp rx=0 tx=0).stop 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 10 monclient: shutdown 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb8c06a940 msgr2=0x7feb8c0ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 --2- 172.21.15.110:0/3428497400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb8c06a940 0x7feb8c0ab900 secure :-1 s=READY pgs=5090 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c3c0 tx=0x7feb8800c880 comp rx=0 tx=0).stop 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb92cd0640 1 -- 172.21.15.110:0/3428497400 reap_dead start 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 shutdown_connections 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 >> 172.21.15.110:0/3428497400 conn(0x7feb8c0b7d70 msgr2=0x7feb8c138cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:22.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 shutdown_connections 2024-01-27T04:20:22.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.850+0000 7feb94733640 1 -- 172.21.15.110:0/3428497400 wait complete. 2024-01-27T04:20:22.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.851+0000 7feb94733640 1 librados: shutdown 2024-01-27T04:20:22.862 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-27T04:20:22.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.961+0000 7f7da4ea2640 10 monclient: get_monmap_and_config 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.961+0000 7f7da4ea2640 10 monclient: build_initial_monmap 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.961+0000 7f7da4ea2640 10 monclient: monmap: 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:22.962788+0000 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:22.962788+0000 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:22.963 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.962+0000 7f7da4ea2640 1 Processor -- start 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.962+0000 7f7da4ea2640 1 -- start start 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.962+0000 7f7da4ea2640 10 monclient: init 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.962+0000 7f7da4ea2640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.962+0000 7f7da4ea2640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.962+0000 7f7da4ea2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7da006f950 0x7f7da006fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 10 monclient(hunting): picked mon.noname-c con 0x7f7da006f950 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da0140850 0x7f7da00b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 10 monclient(hunting): picked mon.noname-b con 0x7f7da0140850 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7da00afbd0 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da0071b80 con 0x7f7da00afbd0 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da0071df0 con 0x7f7da006f950 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da00720d0 con 0x7f7da0140850 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7da4ea2640 10 monclient(hunting): authenticate will time out at 3706.317034s 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7d97fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da0140850 0x7f7da00b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7d9e59c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7da006f950 0x7f7da006fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7d9ed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7da00afbd0 0x7f7da01433a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48080/0 (socket says 172.21.15.110:48080) 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.963+0000 7f7d9ed9d640 1 -- 172.21.15.110:0/2447089051 learned_addr learned my addr 172.21.15.110:0/2447089051 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d9e59c640 10 monclient(hunting): get_auth_request con 0x7f7da006f950 auth_method 0 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d9e59c640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d97fff640 10 monclient(hunting): get_auth_request con 0x7f7da0140850 auth_method 0 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d97fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.964+0000 7f7d97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.966+0000 7f7d9e59c640 10 monclient(hunting): handle_auth_done global_id 20935 payload 274 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.966+0000 7f7d9e59c640 1 -- 172.21.15.110:0/2447089051 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da0140850 msgr2=0x7f7da00b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.966+0000 7f7d9e59c640 1 --2- 172.21.15.110:0/2447089051 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da0140850 0x7f7da00b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.966+0000 7f7d9e59c640 1 -- 172.21.15.110:0/2447089051 >> v1:172.21.15.110:6789/0 conn(0x7f7da00afbd0 legacy=0x7f7da01433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.966+0000 7f7d9e59c640 10 monclient: _finish_hunting 0 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.966+0000 7f7d9e59c640 1 monclient: found mon.noname-c 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.966+0000 7f7d9e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.966+0000 7f7d9e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.966+0000 7f7d9e59c640 1 -- 172.21.15.110:0/2447089051 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da00b0a10 con 0x7f7da006f950 2024-01-27T04:20:22.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.967+0000 7f7d9e59c640 1 --2- 172.21.15.110:0/2447089051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da006f950 0x7f7da006fd30 secure :-1 s=READY pgs=5091 cs=0 l=1 rev1=1 crypto rx=0x7f7d900097a0 tx=0x7f7d9002b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1832f16990431f78 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:22.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 1 -- 172.21.15.110:0/2447089051 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90030800 con 0x7f7da006f950 2024-01-27T04:20:22.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:22.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 10 monclient: dump: 2024-01-27T04:20:22.974 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.974 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.974 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.974 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 10 monclient: _finish_auth 0 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 10 monclient: _check_auth_tickets 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:22.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 1 -- 172.21.15.110:0/2447089051 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d90030960 con 0x7f7da006f950 2024-01-27T04:20:22.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:22.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 1 -- 172.21.15.110:0/2447089051 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90030cf0 con 0x7f7da006f950 2024-01-27T04:20:22.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7da4ea2640 5 monclient: authenticate success, global_id 20935 2024-01-27T04:20:22.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7da4ea2640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:22.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7da4ea2640 4 set_mon_vals no callback set 2024-01-27T04:20:22.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:22.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.971+0000 7f7d9dd9b640 10 monclient: dump: 2024-01-27T04:20:22.976 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 10 monclient: shutdown 2024-01-27T04:20:22.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:22.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 1 -- 172.21.15.110:0/2447089051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da006f950 msgr2=0x7f7da006fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 1 --2- 172.21.15.110:0/2447089051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da006f950 0x7f7da006fd30 secure :-1 s=READY pgs=5091 cs=0 l=1 rev1=1 crypto rx=0x7f7d900097a0 tx=0x7f7d9002b940 comp rx=0 tx=0).stop 2024-01-27T04:20:22.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 1 -- 172.21.15.110:0/2447089051 shutdown_connections 2024-01-27T04:20:22.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 1 --2- 172.21.15.110:0/2447089051 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da0140850 0x7f7da00b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 1 --2- 172.21.15.110:0/2447089051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da006f950 0x7f7da006fd30 unknown :-1 s=CLOSED pgs=5091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 1 -- 172.21.15.110:0/2447089051 >> 172.21.15.110:0/2447089051 conn(0x7f7da00b25d0 msgr2=0x7f7da00b29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:22.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 1 -- 172.21.15.110:0/2447089051 shutdown_connections 2024-01-27T04:20:22.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.972+0000 7f7da4ea2640 1 -- 172.21.15.110:0/2447089051 wait complete. 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.973+0000 7f7da4ea2640 10 monclient: build_initial_monmap 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.973+0000 7f7da4ea2640 10 monclient: monmap: 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:22.983 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:22.984 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:22.984 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.973+0000 7f7da4ea2640 1 librados: starting msgr at 2024-01-27T04:20:22.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.973+0000 7f7da4ea2640 1 librados: starting objecter 2024-01-27T04:20:22.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.973+0000 7f7da4ea2640 1 Processor -- start 2024-01-27T04:20:22.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.973+0000 7f7da4ea2640 1 -- start start 2024-01-27T04:20:22.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.973+0000 7f7da4ea2640 1 librados: setting wanted keys 2024-01-27T04:20:22.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.973+0000 7f7da4ea2640 1 librados: calling monclient init 2024-01-27T04:20:22.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.973+0000 7f7da4ea2640 10 monclient: init 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da006f950 0x7f7da0139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient(hunting): picked mon.noname-c con 0x7f7da006f950 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da00afbd0 0x7f7da01d8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient(hunting): picked mon.noname-b con 0x7f7da00afbd0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0140850 0x7f7da01cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient(hunting): picked mon.noname-a con 0x7f7da0140850 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da01391c0 con 0x7f7da0140850 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da0136ed0 con 0x7f7da00afbd0 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da0139410 con 0x7f7da006f950 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7da4ea2640 10 monclient(hunting): authenticate will time out at 3706.328321s 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7d9e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da006f950 0x7f7da0139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.974+0000 7f7d9ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0140850 0x7f7da01cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d97fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da00afbd0 0x7f7da01d8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:22.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9ed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0140850 0x7f7da01cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44926/0 (socket says 172.21.15.110:44926) 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da006f950 0x7f7da0139ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40724/0 (socket says 172.21.15.110:40724) 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9ed9d640 1 -- 172.21.15.110:0/664823263 learned_addr learned my addr 172.21.15.110:0/664823263 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9ed9d640 10 monclient(hunting): get_auth_request con 0x7f7da0140850 auth_method 0 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9e59c640 10 monclient(hunting): get_auth_request con 0x7f7da006f950 auth_method 0 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9e59c640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d97fff640 10 monclient(hunting): get_auth_request con 0x7f7da00afbd0 auth_method 0 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d97fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.975+0000 7f7d97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 10 monclient(hunting): handle_auth_done global_id 20938 payload 824 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 1 -- 172.21.15.110:0/664823263 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da006f950 msgr2=0x7f7da0139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da006f950 0x7f7da0139ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 1 -- 172.21.15.110:0/664823263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0140850 msgr2=0x7f7da01cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0140850 0x7f7da01cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 10 monclient: _finish_hunting 0 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 1 monclient: found mon.noname-b 2024-01-27T04:20:22.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:22.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.978+0000 7f7d97fff640 1 -- 172.21.15.110:0/664823263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d90009400 con 0x7f7da00afbd0 2024-01-27T04:20:22.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.980+0000 7f7d97fff640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da00afbd0 0x7f7da01d8b20 secure :-1 s=READY pgs=5092 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800e870 tx=0x7f7d8800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:22.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.980+0000 7f7d96ffd640 10 client.?.objecter ms_handle_connect 0x7f7da00afbd0 2024-01-27T04:20:22.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.980+0000 7f7d96ffd640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:22.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 1 -- 172.21.15.110:0/664823263 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8800ee60 con 0x7f7da00afbd0 2024-01-27T04:20:22.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:22.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 10 monclient: dump: 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 10 monclient: _finish_auth 0 2024-01-27T04:20:22.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 10 monclient: _check_auth_tickets 2024-01-27T04:20:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 1 -- 172.21.15.110:0/664823263 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8800c940 con 0x7f7da00afbd0 2024-01-27T04:20:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7da4ea2640 5 monclient: authenticate success, global_id 20938 2024-01-27T04:20:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7da4ea2640 10 monclient: _renew_subs 2024-01-27T04:20:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7da4ea2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da01338a0 con 0x7f7da00afbd0 2024-01-27T04:20:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 1 -- 172.21.15.110:0/664823263 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8800ccd0 con 0x7f7da00afbd0 2024-01-27T04:20:22.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:22.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:22.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d96ffd640 10 monclient: dump: 2024-01-27T04:20:22.995 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:22.995 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:22.995 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.995 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:22.995 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:22.995 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:22.995 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:22.996 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:22.996 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:22.996 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:22.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7d9d59a640 4 set_mon_vals no callback set 2024-01-27T04:20:22.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7da4ea2640 10 client.20938.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:22.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7da4ea2640 10 monclient: _renew_subs 2024-01-27T04:20:22.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7da4ea2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da0137910 con 0x7f7da00afbd0 2024-01-27T04:20:22.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.984+0000 7f7da4ea2640 1 librados: init done 2024-01-27T04:20:22.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.985+0000 7f7da4ea2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:22.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.985+0000 7f7da4ea2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:22.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.985+0000 7f7da4ea2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:22.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:22.985+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da004ea50 con 0x7f7da00afbd0 2024-01-27T04:20:23.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.003+0000 7f7d96ffd640 1 -- 172.21.15.110:0/664823263 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f7d88013470 con 0x7f7da00afbd0 2024-01-27T04:20:23.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.004+0000 7f7d96ffd640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7d8006c9a0 0x7f7d8006ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.004+0000 7f7d9e59c640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7d8006c9a0 0x7f7d8006ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.004+0000 7f7d9e59c640 10 monclient: get_auth_request con 0x7f7d8006c9a0 auth_method 0 2024-01-27T04:20:23.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.005+0000 7f7d96ffd640 10 client.20938.objecter ms_handle_connect 0x7f7d8006c9a0 2024-01-27T04:20:23.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.005+0000 7f7d9e59c640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7d8006c9a0 0x7f7d8006ee60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7d90036000 tx=0x7f7d90040040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:23.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.006+0000 7f7d96ffd640 1 -- 172.21.15.110:0/664823263 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f7d8800ce30 con 0x7f7da00afbd0 2024-01-27T04:20:23.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.006+0000 7f7d96ffd640 10 client.20938.objecter ms_dispatch 0x7f7da0000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:23.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.006+0000 7f7d96ffd640 3 client.20938.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:23.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.006+0000 7f7d96ffd640 3 client.20938.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:23.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.006+0000 7f7d96ffd640 20 client.20938.objecter dump_active .. 0 homeless 2024-01-27T04:20:23.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.242+0000 7f7d96ffd640 1 -- 172.21.15.110:0/664823263 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f7d88054e50 con 0x7f7da00afbd0 2024-01-27T04:20:23.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.242+0000 7f7d96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:23.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.242+0000 7f7d96ffd640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:23.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.418+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7da0139580 con 0x7f7d8006c9a0 2024-01-27T04:20:23.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.418+0000 7f7d96ffd640 1 -- 172.21.15.110:0/664823263 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7da0139580 con 0x7f7d8006c9a0 2024-01-27T04:20:23.420 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 2 to deep-scrub 2024-01-27T04:20:23.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 10 librados: watch_flush enter 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 10 librados: watch_flush exit 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 20 client.20938.objecter shutdown clearing up homeless session... 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 10 client.20938.objecter successfully canceled tick 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7d8006c9a0 msgr2=0x7f7d8006ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7d8006c9a0 0x7f7d8006ee60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7d90036000 tx=0x7f7d90040040 comp rx=0 tx=0).stop 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 10 monclient: shutdown 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da00afbd0 msgr2=0x7f7da01d8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da00afbd0 0x7f7da01d8b20 secure :-1 s=READY pgs=5092 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800e870 tx=0x7f7d8800ed30 comp rx=0 tx=0).stop 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 shutdown_connections 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7d8006c9a0 0x7f7d8006ee60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7da0140850 0x7f7da01cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da00afbd0 0x7f7da01d8b20 unknown :-1 s=CLOSED pgs=5092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 --2- 172.21.15.110:0/664823263 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7da006f950 0x7f7da0139ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 >> 172.21.15.110:0/664823263 conn(0x7f7da00b25d0 msgr2=0x7f7da013e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:23.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 shutdown_connections 2024-01-27T04:20:23.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 -- 172.21.15.110:0/664823263 wait complete. 2024-01-27T04:20:23.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.421+0000 7f7da4ea2640 1 librados: shutdown 2024-01-27T04:20:23.434 INFO:tasks.ceph:Scrubbing osd.3 2024-01-27T04:20:23.435 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-27T04:20:23.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.537+0000 7f629e77d640 10 monclient: get_monmap_and_config 2024-01-27T04:20:23.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.537+0000 7f629e77d640 10 monclient: build_initial_monmap 2024-01-27T04:20:23.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.537+0000 7f629e77d640 10 monclient: monmap: 2024-01-27T04:20:23.538 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:23.538 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.538 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:23.538324+0000 2024-01-27T04:20:23.538 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:23.538324+0000 2024-01-27T04:20:23.538 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:23.539 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.539 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:23.539 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:23.539 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:23.539 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:23.539 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:23.539 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.537+0000 7f629e77d640 1 Processor -- start 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.537+0000 7f629e77d640 1 -- start start 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.537+0000 7f629e77d640 10 monclient: init 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f629806a940 0x7f62981426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 10 monclient(hunting): picked mon.noname-a con 0x7f629806a940 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62980a9e40 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298142c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 10 monclient(hunting): picked mon.noname-b con 0x7f62980b3450 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62980b3860 con 0x7f62980a9e40 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:23.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629806ad20 con 0x7f629806a940 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62980aac50 con 0x7f62980b3450 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.538+0000 7f629e77d640 10 monclient(hunting): authenticate will time out at 3706.892525s 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f6297fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f629806a940 0x7f62981426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f629cd1a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f6297fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f629806a940 0x7f62981426d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44930/0 (socket says 172.21.15.110:44930) 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f6297fff640 1 -- 172.21.15.110:0/205103782 learned_addr learned my addr 172.21.15.110:0/205103782 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f6297fff640 10 monclient(hunting): get_auth_request con 0x7f629806a940 auth_method 0 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f6297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f6297fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f6297fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f629cd1a640 10 monclient(hunting): get_auth_request con 0x7f62980b3450 auth_method 0 2024-01-27T04:20:23.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f629cd1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:23.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f629cd1a640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.539+0000 7f629cd1a640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.540+0000 7f629cd1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:23.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.540+0000 7f629cd1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:23.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.540+0000 7f629cd1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:23.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.540+0000 7f6297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:23.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.540+0000 7f6297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:23.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.540+0000 7f6297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:23.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.541+0000 7f629cd1a640 10 monclient(hunting): handle_auth_done global_id 18902 payload 274 2024-01-27T04:20:23.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.541+0000 7f629cd1a640 1 -- 172.21.15.110:0/205103782 >> v2:172.21.15.110:3300/0 conn(0x7f629806a940 msgr2=0x7f62981426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:23.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.541+0000 7f629cd1a640 1 --2- 172.21.15.110:0/205103782 >> v2:172.21.15.110:3300/0 conn(0x7f629806a940 0x7f62981426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.541+0000 7f629cd1a640 1 -- 172.21.15.110:0/205103782 >> v1:172.21.15.110:6789/0 conn(0x7f62980a9e40 legacy=0x7f62980b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.541+0000 7f629cd1a640 10 monclient: _finish_hunting 0 2024-01-27T04:20:23.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.541+0000 7f629cd1a640 1 monclient: found mon.noname-b 2024-01-27T04:20:23.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.541+0000 7f629cd1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:23.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.541+0000 7f629cd1a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:23.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.541+0000 7f629cd1a640 1 -- 172.21.15.110:0/205103782 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62980aaf30 con 0x7f62980b3450 2024-01-27T04:20:23.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.544+0000 7f629cd1a640 1 --2- 172.21.15.110:0/205103782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298142c10 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 crypto rx=0x7f6290009d90 tx=0x7f629002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7071c753b9e51438 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 1 -- 172.21.15.110:0/205103782 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629002cc50 con 0x7f62980b3450 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 10 monclient: dump: 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.549 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 10 monclient: _finish_auth 0 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 10 monclient: _check_auth_tickets 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 1 -- 172.21.15.110:0/205103782 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629002cdb0 con 0x7f62980b3450 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 1 -- 172.21.15.110:0/205103782 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290034660 con 0x7f62980b3450 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f6296ffd640 10 monclient: dump: 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:23.550 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f629e77d640 5 monclient: authenticate success, global_id 18902 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f629e77d640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:23.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.547+0000 7f629e77d640 4 set_mon_vals no callback set 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 10 monclient: shutdown 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 1 -- 172.21.15.110:0/205103782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 msgr2=0x7f6298142c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 1 --2- 172.21.15.110:0/205103782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298142c10 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 crypto rx=0x7f6290009d90 tx=0x7f629002bea0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 1 -- 172.21.15.110:0/205103782 shutdown_connections 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 1 --2- 172.21.15.110:0/205103782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298142c10 unknown :-1 s=CLOSED pgs=3413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 1 --2- 172.21.15.110:0/205103782 >> v2:172.21.15.110:3300/0 conn(0x7f629806a940 0x7f62981426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 1 -- 172.21.15.110:0/205103782 >> 172.21.15.110:0/205103782 conn(0x7f62980b7d70 msgr2=0x7f62980b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 1 -- 172.21.15.110:0/205103782 shutdown_connections 2024-01-27T04:20:23.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.548+0000 7f629e77d640 1 -- 172.21.15.110:0/205103782 wait complete. 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.549+0000 7f629e77d640 10 monclient: build_initial_monmap 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.549+0000 7f629e77d640 10 monclient: monmap: 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:23.556 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.549+0000 7f629e77d640 1 librados: starting msgr at 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.549+0000 7f629e77d640 1 librados: starting objecter 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.549+0000 7f629e77d640 1 Processor -- start 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.549+0000 7f629e77d640 1 -- start start 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.549+0000 7f629e77d640 1 librados: setting wanted keys 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.549+0000 7f629e77d640 1 librados: calling monclient init 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.549+0000 7f629e77d640 10 monclient: init 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T04:20:23.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f629806a940 0x7f6298146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient(hunting): picked mon.noname-b con 0x7f629806a940 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62980a9e40 0x7f62981dad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient(hunting): picked mon.noname-a con 0x7f62980a9e40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298137f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient(hunting): picked mon.noname-c con 0x7f62980b3450 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6298148090 con 0x7f62980a9e40 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62981da870 con 0x7f629806a940 2024-01-27T04:20:23.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629813c270 con 0x7f62980b3450 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629e77d640 10 monclient(hunting): authenticate will time out at 3706.904473s 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f629cd1a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298137f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.550+0000 7f62977fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62980a9e40 0x7f62981dad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f6297fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f629806a940 0x7f6298146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f629cd1a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298137f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40754/0 (socket says 172.21.15.110:40754) 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f629cd1a640 1 -- 172.21.15.110:0/3276897988 learned_addr learned my addr 172.21.15.110:0/3276897988 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f629cd1a640 10 monclient(hunting): get_auth_request con 0x7f62980b3450 auth_method 0 2024-01-27T04:20:23.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f629cd1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f629cd1a640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f629cd1a640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f6297fff640 10 monclient(hunting): get_auth_request con 0x7f629806a940 auth_method 0 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f6297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f6297fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f6297fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f629cd1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f629cd1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.551+0000 7f629cd1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.552+0000 7f6297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:23.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.552+0000 7f6297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:23.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.552+0000 7f6297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:23.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.552+0000 7f62977fe640 10 monclient(hunting): get_auth_request con 0x7f62980a9e40 auth_method 0 2024-01-27T04:20:23.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.552+0000 7f62977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:23.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.552+0000 7f62977fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.552+0000 7f62977fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.553+0000 7f62977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:23.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.553+0000 7f62977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:23.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.553+0000 7f62977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:23.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 10 monclient(hunting): handle_auth_done global_id 18905 payload 824 2024-01-27T04:20:23.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 1 -- 172.21.15.110:0/3276897988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f629806a940 msgr2=0x7f6298146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f629806a940 0x7f6298146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 1 -- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62980a9e40 msgr2=0x7f62981dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62980a9e40 0x7f62981dad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 10 monclient: _finish_hunting 0 2024-01-27T04:20:23.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 1 monclient: found mon.noname-c 2024-01-27T04:20:23.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:23.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T04:20:23.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.554+0000 7f629cd1a640 1 -- 172.21.15.110:0/3276897988 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290009a70 con 0x7f62980b3450 2024-01-27T04:20:23.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.559+0000 7f629cd1a640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298137f20 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7f6290009d90 tx=0x7f6290004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:23.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.559+0000 7f6294ff9640 10 client.?.objecter ms_handle_connect 0x7f62980b3450 2024-01-27T04:20:23.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.559+0000 7f6294ff9640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:23.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 1 -- 172.21.15.110:0/3276897988 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290040650 con 0x7f62980b3450 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 10 monclient: dump: 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:23.566 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 10 monclient: _finish_auth 0 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 10 monclient: _check_auth_tickets 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f629e77d640 5 monclient: authenticate success, global_id 18905 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 1 -- 172.21.15.110:0/3276897988 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62900407b0 con 0x7f62980b3450 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 1 -- 172.21.15.110:0/3276897988 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290040b60 con 0x7f62980b3450 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f6294ff9640 10 monclient: dump: 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.567 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f629e77d640 10 monclient: _renew_subs 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f62967fc640 4 set_mon_vals no callback set 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.563+0000 7f629e77d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629813c4a0 con 0x7f62980b3450 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 10 client.18905.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 10 monclient: _renew_subs 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:23.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629813f1f0 con 0x7f62980b3450 2024-01-27T04:20:23.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 1 librados: init done 2024-01-27T04:20:23.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 10 client.18905.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:23.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 15 client.18905.objecter _session_command_op_assign -1 1 2024-01-27T04:20:23.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 20 client.18905.objecter _get_session osd=-1 returning homeless 2024-01-27T04:20:23.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 10 client.18905.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:23.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:23.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.564+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62981da6d0 con 0x7f62980b3450 2024-01-27T04:20:23.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.589+0000 7f6294ff9640 1 -- 172.21.15.110:0/3276897988 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f6290040cc0 con 0x7f62980b3450 2024-01-27T04:20:23.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.590+0000 7f6294ff9640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f628406c9d0 0x7f628406ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.590+0000 7f6297fff640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f628406c9d0 0x7f628406ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.590+0000 7f6297fff640 10 monclient: get_auth_request con 0x7f628406c9d0 auth_method 0 2024-01-27T04:20:23.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.591+0000 7f6297fff640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f628406c9d0 0x7f628406ee90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6288009b30 tx=0x7f6288009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:23.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.591+0000 7f6294ff9640 10 client.18905.objecter ms_handle_connect 0x7f628406c9d0 2024-01-27T04:20:23.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 1 -- 172.21.15.110:0/3276897988 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f629007b550 con 0x7f62980b3450 2024-01-27T04:20:23.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 10 client.18905.objecter ms_dispatch 0x7f6298000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:23.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 3 client.18905.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:23.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 3 client.18905.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 10 client.18905.objecter checking command 1 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f6284074750 0x7f6284076bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 20 client.18905.objecter _get_session s=0x7f6284074080 osd=3 3 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 20 client.18905.objecter put_session s=0x7f6284074080 osd=3 3 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 15 client.18905.objecter _session_command_op_remove -1 1 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 20 client.18905.objecter _get_session s=0x7f6284074080 osd=3 3 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 20 client.18905.objecter get_session s=0x7f6284074080 osd=3 3 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 15 client.18905.objecter _session_command_op_assign 3 1 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 20 client.18905.objecter put_session s=0x7f6284074080 osd=3 4 2024-01-27T04:20:23.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 10 client.18905.objecter _send_command 1 2024-01-27T04:20:23.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 1 -- 172.21.15.110:0/3276897988 --> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62840772c0 con 0x7f6284074750 2024-01-27T04:20:23.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 20 client.18905.objecter dump_active .. 0 homeless 2024-01-27T04:20:23.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 1 -- 172.21.15.110:0/3276897988 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=1334) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62900777f0 con 0x7f62980b3450 2024-01-27T04:20:23.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.593+0000 7f6294ff9640 10 monclient: handle_get_version_reply finishing 1 version 1334 2024-01-27T04:20:23.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.594+0000 7f62977fe640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f6284074750 0x7f6284076bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.594+0000 7f62977fe640 10 monclient: get_auth_request con 0x7f6284074750 auth_method 0 2024-01-27T04:20:23.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.597+0000 7f6294ff9640 10 client.18905.objecter ms_handle_connect 0x7f6284074750 2024-01-27T04:20:23.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.597+0000 7f62977fe640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f6284074750 0x7f6284076bf0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:23.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.628+0000 7f6294ff9640 1 -- 172.21.15.110:0/3276897988 <== osd.3 v2:172.21.15.110:6800/2374111370 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f62840772c0 con 0x7f6284074750 2024-01-27T04:20:23.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.628+0000 7f6294ff9640 10 client.18905.objecter ms_dispatch 0x7f6298000b90 command_reply(tid 1: 0 ) v1 2024-01-27T04:20:23.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.628+0000 7f6294ff9640 10 client.18905.objecter _finish_command 1 = system:0 2024-01-27T04:20:23.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.628+0000 7f6294ff9640 20 client.18905.objecter put_session s=0x7f6284074080 osd=3 4 2024-01-27T04:20:23.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.628+0000 7f6294ff9640 15 client.18905.objecter _session_command_op_remove 3 1 2024-01-27T04:20:23.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 10 client.18905.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-27T04:20:23.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 15 client.18905.objecter _session_command_op_assign -1 2 2024-01-27T04:20:23.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 20 client.18905.objecter _get_session s=0x7f6284074080 osd=3 3 2024-01-27T04:20:23.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 20 client.18905.objecter put_session s=0x7f6284074080 osd=3 3 2024-01-27T04:20:23.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 20 client.18905.objecter _get_session s=0x7f6284074080 osd=3 3 2024-01-27T04:20:23.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 15 client.18905.objecter _session_command_op_remove -1 2 2024-01-27T04:20:23.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 20 client.18905.objecter get_session s=0x7f6284074080 osd=3 3 2024-01-27T04:20:23.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 15 client.18905.objecter _session_command_op_assign 3 2 2024-01-27T04:20:23.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 20 client.18905.objecter put_session s=0x7f6284074080 osd=3 4 2024-01-27T04:20:23.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 10 client.18905.objecter _send_command 2 2024-01-27T04:20:23.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.655+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 --> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6298074d10 con 0x7f6284074750 2024-01-27T04:20:23.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f6294ff9640 1 -- 172.21.15.110:0/3276897988 <== osd.3 v2:172.21.15.110:6800/2374111370 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6298074d10 con 0x7f6284074750 2024-01-27T04:20:23.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f6294ff9640 10 client.18905.objecter ms_dispatch 0x7f6298000b90 command_reply(tid 2: 0 ) v1 2024-01-27T04:20:23.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f6294ff9640 10 client.18905.objecter _finish_command 2 = system:0 2024-01-27T04:20:23.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f6294ff9640 20 client.18905.objecter put_session s=0x7f6284074080 osd=3 4 2024-01-27T04:20:23.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f6294ff9640 15 client.18905.objecter _session_command_op_remove 3 2 2024-01-27T04:20:23.671 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-01-27T04:20:23.671 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-27T04:20:23.671 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-01-27T04:20:23.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 10 librados: watch_flush enter 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 10 librados: watch_flush exit 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 10 client.18905.objecter close_session for osd.3 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f6284074750 msgr2=0x7f6284076bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:6800/2374111370,v1:172.21.15.110:6801/2374111370] conn(0x7f6284074750 0x7f6284076bf0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 20 client.18905.objecter put_session s=0x7f6284074080 osd=3 1 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 20 client.18905.objecter shutdown clearing up homeless session... 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 10 client.18905.objecter successfully canceled tick 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f628406c9d0 msgr2=0x7f628406ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f628406c9d0 0x7f628406ee90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6288009b30 tx=0x7f6288009240 comp rx=0 tx=0).stop 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 10 monclient: shutdown 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.669+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 msgr2=0x7f6298137f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.670+0000 7f629e77d640 1 --2- 172.21.15.110:0/3276897988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62980b3450 0x7f6298137f20 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7f6290009d90 tx=0x7f6290004a40 comp rx=0 tx=0).stop 2024-01-27T04:20:23.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.670+0000 7f629cd1a640 1 -- 172.21.15.110:0/3276897988 reap_dead start 2024-01-27T04:20:23.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.670+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 shutdown_connections 2024-01-27T04:20:23.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.670+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 >> 172.21.15.110:0/3276897988 conn(0x7f62980b7d70 msgr2=0x7f6298139440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:23.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.670+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 shutdown_connections 2024-01-27T04:20:23.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.670+0000 7f629e77d640 1 -- 172.21.15.110:0/3276897988 wait complete. 2024-01-27T04:20:23.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.670+0000 7f629e77d640 1 librados: shutdown 2024-01-27T04:20:23.681 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-27T04:20:23.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.788+0000 7fe9a5864640 10 monclient: get_monmap_and_config 2024-01-27T04:20:23.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.788+0000 7fe9a5864640 10 monclient: build_initial_monmap 2024-01-27T04:20:23.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.788+0000 7fe9a5864640 10 monclient: monmap: 2024-01-27T04:20:23.789 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:23.789 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.789 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:23.789577+0000 2024-01-27T04:20:23.789 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:23.789577+0000 2024-01-27T04:20:23.790 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:23.790 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.790 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:23.790 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:23.790 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:23.790 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:23.790 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:23.790 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.789+0000 7fe9a5864640 1 Processor -- start 2024-01-27T04:20:23.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.789+0000 7fe9a5864640 1 -- start start 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.789+0000 7fe9a5864640 10 monclient: init 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9a006fc00 0x7fe9a013d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 10 monclient(hunting): picked mon.noname-a con 0x7fe9a006fc00 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9a013d7a0 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9a0140ac0 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00b0460 con 0x7fe9a013d7a0 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a0071b70 con 0x7fe9a0140ac0 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a013dbe0 con 0x7fe9a006fc00 2024-01-27T04:20:23.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:23.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe9a5864640 10 monclient(hunting): authenticate will time out at 3707.144106s 2024-01-27T04:20:23.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.790+0000 7fe99effd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9a006fc00 0x7fe9a013d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.791+0000 7fe99f7fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe9a0140ac0 0x7fe9a014a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:34618/0 (socket says 172.21.15.110:34618) 2024-01-27T04:20:23.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.791+0000 7fe99f7fe640 1 -- 172.21.15.110:0/3275386932 learned_addr learned my addr 172.21.15.110:0/3275386932 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T04:20:23.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.791+0000 7fe99effd640 10 monclient(hunting): get_auth_request con 0x7fe9a006fc00 auth_method 0 2024-01-27T04:20:23.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.791+0000 7fe99effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:23.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.791+0000 7fe99effd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.791+0000 7fe99effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.792+0000 7fe99effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:23.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.792+0000 7fe99effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:23.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.792+0000 7fe99effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:23.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.793+0000 7fe99e7fc640 1 -- 172.21.15.110:0/3275386932 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478519406 0 0) 0x7fe9a0071b70 con 0x7fe9a0140ac0 2024-01-27T04:20:23.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.793+0000 7fe99e7fc640 10 monclient(hunting): _negotiate 2024-01-27T04:20:23.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.793+0000 7fe99e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.793+0000 7fe99e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.793+0000 7fe99e7fc640 10 monclient(hunting): my global_id is 20944 2024-01-27T04:20:23.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.793+0000 7fe99e7fc640 1 -- 172.21.15.110:0/3275386932 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c003520 con 0x7fe9a0140ac0 2024-01-27T04:20:23.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.794+0000 7fe99effd640 10 monclient(hunting): handle_auth_done global_id 17034 payload 274 2024-01-27T04:20:23.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.794+0000 7fe99effd640 1 -- 172.21.15.110:0/3275386932 >> v1:172.21.15.162:6789/0 conn(0x7fe9a0140ac0 legacy=0x7fe9a014a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.794+0000 7fe99effd640 1 -- 172.21.15.110:0/3275386932 >> v1:172.21.15.110:6789/0 conn(0x7fe9a013d7a0 legacy=0x7fe9a01403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.794+0000 7fe99effd640 10 monclient: _finish_hunting 0 2024-01-27T04:20:23.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.794+0000 7fe99effd640 1 monclient: found mon.noname-a 2024-01-27T04:20:23.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.794+0000 7fe99effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:23.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.794+0000 7fe99effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T04:20:23.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.794+0000 7fe99effd640 1 -- 172.21.15.110:0/3275386932 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a0071e50 con 0x7fe9a006fc00 2024-01-27T04:20:23.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.796+0000 7fe99effd640 1 --2- 172.21.15.110:0/3275386932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a006fc00 0x7fe9a013d260 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7fe9900097a0 tx=0x7fe99002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d479c9bbbd78931 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:23.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 1 -- 172.21.15.110:0/3275386932 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe990030610 con 0x7fe9a006fc00 2024-01-27T04:20:23.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:23.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T04:20:23.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 10 monclient: dump: 2024-01-27T04:20:23.801 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 10 monclient: _finish_auth 0 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 10 monclient: _check_auth_tickets 2024-01-27T04:20:23.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 1 -- 172.21.15.110:0/3275386932 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe990030770 con 0x7fe9a006fc00 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe9a5864640 5 monclient: authenticate success, global_id 17034 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe9a5864640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe9a5864640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 1 -- 172.21.15.110:0/3275386932 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe990030b00 con 0x7fe9a006fc00 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe9a5864640 4 set_mon_vals no callback set 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.800+0000 7fe99e7fc640 10 monclient: dump: 2024-01-27T04:20:23.803 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:23.804 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:23.804 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.801+0000 7fe9a5864640 10 monclient: shutdown 2024-01-27T04:20:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.801+0000 7fe9a5864640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.801+0000 7fe9a5864640 1 -- 172.21.15.110:0/3275386932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a006fc00 msgr2=0x7fe9a013d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.801+0000 7fe9a5864640 1 --2- 172.21.15.110:0/3275386932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a006fc00 0x7fe9a013d260 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7fe9900097a0 tx=0x7fe99002b960 comp rx=0 tx=0).stop 2024-01-27T04:20:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.801+0000 7fe9a5864640 1 -- 172.21.15.110:0/3275386932 shutdown_connections 2024-01-27T04:20:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.801+0000 7fe9a5864640 1 --2- 172.21.15.110:0/3275386932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a006fc00 0x7fe9a013d260 unknown :-1 s=CLOSED pgs=3469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.801+0000 7fe9a5864640 1 -- 172.21.15.110:0/3275386932 >> 172.21.15.110:0/3275386932 conn(0x7fe9a0066f00 msgr2=0x7fe9a00b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.801+0000 7fe9a5864640 1 -- 172.21.15.110:0/3275386932 shutdown_connections 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.801+0000 7fe9a5864640 1 -- 172.21.15.110:0/3275386932 wait complete. 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.802+0000 7fe9a5864640 10 monclient: build_initial_monmap 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.802+0000 7fe9a5864640 10 monclient: monmap: 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:23.810 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.802+0000 7fe9a5864640 1 librados: starting msgr at 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.802+0000 7fe9a5864640 1 librados: starting objecter 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.802+0000 7fe9a5864640 1 Processor -- start 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.802+0000 7fe9a5864640 1 -- start start 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.802+0000 7fe9a5864640 1 librados: setting wanted keys 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.802+0000 7fe9a5864640 1 librados: calling monclient init 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.802+0000 7fe9a5864640 10 monclient: init 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:23.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a006fc00 0x7fe9a00b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a006fc00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a013d7a0 0x7fe9a013f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient(hunting): picked mon.noname-a con 0x7fe9a013d7a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a0140ac0 0x7fe9a013c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a0140ac0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a013fde0 con 0x7fe9a013d7a0 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a013dee0 con 0x7fe9a0140ac0 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a013e370 con 0x7fe9a006fc00 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:23.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.803+0000 7fe9a5864640 10 monclient(hunting): authenticate will time out at 3707.157486s 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe9967fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a013d7a0 0x7fe9a013f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99effd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a006fc00 0x7fe9a00b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a0140ac0 0x7fe9a013c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe9967fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a013d7a0 0x7fe9a013f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44950/0 (socket says 172.21.15.110:44950) 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe9967fc640 1 -- 172.21.15.110:0/2209222347 learned_addr learned my addr 172.21.15.110:0/2209222347 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe9967fc640 10 monclient(hunting): get_auth_request con 0x7fe9a013d7a0 auth_method 0 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe9967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe9967fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe9967fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99f7fe640 10 monclient(hunting): get_auth_request con 0x7fe9a0140ac0 auth_method 0 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:23.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99effd640 10 monclient(hunting): get_auth_request con 0x7fe9a006fc00 auth_method 0 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99effd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.804+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.805+0000 7fe9967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.805+0000 7fe9967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.805+0000 7fe9967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.805+0000 7fe99effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:23.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.805+0000 7fe99effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:23.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.805+0000 7fe99effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:23.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_done global_id 20947 payload 824 2024-01-27T04:20:23.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 1 -- 172.21.15.110:0/2209222347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a006fc00 msgr2=0x7fe9a00b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a006fc00 0x7fe9a00b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 1 -- 172.21.15.110:0/2209222347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a013d7a0 msgr2=0x7fe9a013f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:23.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a013d7a0 0x7fe9a013f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:23.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 10 monclient: _finish_hunting 0 2024-01-27T04:20:23.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 1 monclient: found mon.noname-b 2024-01-27T04:20:23.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:23.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:23.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.807+0000 7fe99f7fe640 1 -- 172.21.15.110:0/2209222347 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe990009400 con 0x7fe9a0140ac0 2024-01-27T04:20:23.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.810+0000 7fe99f7fe640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a0140ac0 0x7fe9a013c600 secure :-1 s=READY pgs=5094 cs=0 l=1 rev1=1 crypto rx=0x7fe99800caa0 tx=0x7fe99800cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:23.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.810+0000 7fe99cff9640 10 client.?.objecter ms_handle_connect 0x7fe9a0140ac0 2024-01-27T04:20:23.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.810+0000 7fe99cff9640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:23.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 1 -- 172.21.15.110:0/2209222347 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99800ab40 con 0x7fe9a0140ac0 2024-01-27T04:20:23.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 10 monclient: dump: 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 10 monclient: _finish_auth 0 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 10 monclient: _check_auth_tickets 2024-01-27T04:20:23.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 1 -- 172.21.15.110:0/2209222347 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99800aca0 con 0x7fe9a0140ac0 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 1 -- 172.21.15.110:0/2209222347 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe998012600 con 0x7fe9a0140ac0 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe9a5864640 5 monclient: authenticate success, global_id 20947 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe9a5864640 10 monclient: _renew_subs 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe9a5864640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a013cb40 con 0x7fe9a0140ac0 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99dffb640 4 set_mon_vals no callback set 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe99cff9640 10 monclient: dump: 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:23.820 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe9a5864640 10 client.20947.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe9a5864640 10 monclient: _renew_subs 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe9a5864640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a00af6e0 con 0x7fe9a0140ac0 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.813+0000 7fe9a5864640 1 librados: init done 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.814+0000 7fe9a5864640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:23.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.814+0000 7fe9a5864640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:23.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.814+0000 7fe9a5864640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:23.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.814+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a013e070 con 0x7fe9a0140ac0 2024-01-27T04:20:23.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.833+0000 7fe99cff9640 1 -- 172.21.15.110:0/2209222347 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fe998007070 con 0x7fe9a0140ac0 2024-01-27T04:20:23.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.833+0000 7fe99cff9640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe98006c9a0 0x7fe98006ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:23.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.833+0000 7fe99effd640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe98006c9a0 0x7fe98006ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:23.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.833+0000 7fe99effd640 10 monclient: get_auth_request con 0x7fe98006c9a0 auth_method 0 2024-01-27T04:20:23.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.834+0000 7fe99effd640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe98006c9a0 0x7fe98006ee60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe990036000 tx=0x7fe990008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:23.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.834+0000 7fe99cff9640 10 client.20947.objecter ms_handle_connect 0x7fe98006c9a0 2024-01-27T04:20:23.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.835+0000 7fe99cff9640 1 -- 172.21.15.110:0/2209222347 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7fe998058680 con 0x7fe9a0140ac0 2024-01-27T04:20:23.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.835+0000 7fe99cff9640 10 client.20947.objecter ms_dispatch 0x7fe9a0000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:23.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.835+0000 7fe99cff9640 3 client.20947.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:23.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.835+0000 7fe99cff9640 3 client.20947.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:23.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:23.835+0000 7fe99cff9640 20 client.20947.objecter dump_active .. 0 homeless 2024-01-27T04:20:24.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.077+0000 7fe99cff9640 1 -- 172.21.15.110:0/2209222347 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fe998054920 con 0x7fe9a0140ac0 2024-01-27T04:20:24.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.077+0000 7fe99cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:24.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.077+0000 7fe99cff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:24.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.256+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9a004ea50 con 0x7fe98006c9a0 2024-01-27T04:20:24.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.257+0000 7fe99cff9640 1 -- 172.21.15.110:0/2209222347 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe9a004ea50 con 0x7fe98006c9a0 2024-01-27T04:20:24.258 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 3 to deep-scrub 2024-01-27T04:20:24.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 10 librados: watch_flush enter 2024-01-27T04:20:24.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 10 librados: watch_flush exit 2024-01-27T04:20:24.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 20 client.20947.objecter shutdown clearing up homeless session... 2024-01-27T04:20:24.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 10 client.20947.objecter successfully canceled tick 2024-01-27T04:20:24.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe98006c9a0 msgr2=0x7fe98006ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe98006c9a0 0x7fe98006ee60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe990036000 tx=0x7fe990008030 comp rx=0 tx=0).stop 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 10 monclient: shutdown 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a0140ac0 msgr2=0x7fe9a013c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a0140ac0 0x7fe9a013c600 secure :-1 s=READY pgs=5094 cs=0 l=1 rev1=1 crypto rx=0x7fe99800caa0 tx=0x7fe99800cea0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 shutdown_connections 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe98006c9a0 0x7fe98006ee60 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a0140ac0 0x7fe9a013c600 unknown :-1 s=CLOSED pgs=5094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a013d7a0 0x7fe9a013f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 --2- 172.21.15.110:0/2209222347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a006fc00 0x7fe9a00b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 >> 172.21.15.110:0/2209222347 conn(0x7fe9a0066f00 msgr2=0x7fe9a00b5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 shutdown_connections 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.259+0000 7fe9a5864640 1 -- 172.21.15.110:0/2209222347 wait complete. 2024-01-27T04:20:24.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.260+0000 7fe9a5864640 1 librados: shutdown 2024-01-27T04:20:24.272 INFO:tasks.ceph:Scrubbing osd.4 2024-01-27T04:20:24.272 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-27T04:20:24.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.374+0000 7f443303f640 10 monclient: get_monmap_and_config 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.374+0000 7f443303f640 10 monclient: build_initial_monmap 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.374+0000 7f443303f640 10 monclient: monmap: 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:24.375973+0000 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:24.375973+0000 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:24.376 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:24.377 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.375+0000 7f443303f640 1 Processor -- start 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.375+0000 7f443303f640 1 -- start start 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.375+0000 7f443303f640 10 monclient: init 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c06bcb0 0x7f442c0ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient(hunting): picked mon.noname-b con 0x7f442c06bcb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f442c06c090 0x7f442c1426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient(hunting): picked mon.noname-c con 0x7f442c06c090 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:24.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f442c142c10 0x7f442c13f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient(hunting): picked mon.noname-a con 0x7f442c142c10 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442c142240 con 0x7f442c142c10 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442c0addb0 con 0x7f442c06c090 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442c0ae090 con 0x7f442c06bcb0 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f443303f640 10 monclient(hunting): authenticate will time out at 3707.730098s 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f4430ddb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c06bcb0 0x7f442c0ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f44315dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f442c142c10 0x7f442c13f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.376+0000 7f4423fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f442c06c090 0x7f442c1426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f44315dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f442c142c10 0x7f442c13f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44954/0 (socket says 172.21.15.110:44954) 2024-01-27T04:20:24.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4430ddb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c06bcb0 0x7f442c0ae4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40762/0 (socket says 172.21.15.110:40762) 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f44315dc640 1 -- 172.21.15.110:0/3493593524 learned_addr learned my addr 172.21.15.110:0/3493593524 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f44315dc640 10 monclient(hunting): get_auth_request con 0x7f442c142c10 auth_method 0 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f44315dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f44315dc640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f44315dc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4430ddb640 10 monclient(hunting): get_auth_request con 0x7f442c06bcb0 auth_method 0 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4430ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4430ddb640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4430ddb640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4423fff640 10 monclient(hunting): get_auth_request con 0x7f442c06c090 auth_method 0 2024-01-27T04:20:24.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4423fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4423fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4423fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4430ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4430ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4430ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f44315dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f44315dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f44315dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4423fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4423fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.377+0000 7f4423fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 10 monclient(hunting): handle_auth_done global_id 18911 payload 274 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 1 -- 172.21.15.110:0/3493593524 >> v2:172.21.15.162:3300/0 conn(0x7f442c06c090 msgr2=0x7f442c1426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 1 --2- 172.21.15.110:0/3493593524 >> v2:172.21.15.162:3300/0 conn(0x7f442c06c090 0x7f442c1426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 1 -- 172.21.15.110:0/3493593524 >> v2:172.21.15.110:3300/0 conn(0x7f442c142c10 msgr2=0x7f442c13f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 1 --2- 172.21.15.110:0/3493593524 >> v2:172.21.15.110:3300/0 conn(0x7f442c142c10 0x7f442c13f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 10 monclient: _finish_hunting 0 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 1 monclient: found mon.noname-b 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4430ddb640 1 -- 172.21.15.110:0/3493593524 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c06d070 con 0x7f442c06bcb0 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f4423fff640 1 --2- 172.21.15.110:0/3493593524 >> v2:172.21.15.162:3300/0 conn(0x7f442c06c090 0x7f442c1426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T04:20:24.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.379+0000 7f44315dc640 1 --2- 172.21.15.110:0/3493593524 >> v2:172.21.15.110:3300/0 conn(0x7f442c142c10 0x7f442c13f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T04:20:24.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.381+0000 7f4430ddb640 1 --2- 172.21.15.110:0/3493593524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c06bcb0 0x7f442c0ae4b0 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7f442c0672e0 tx=0x7f442802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e04fe489b92f7ad server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:24.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 1 -- 172.21.15.110:0/3493593524 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442802e070 con 0x7f442c06bcb0 2024-01-27T04:20:24.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:24.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 10 monclient: dump: 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 10 monclient: _finish_auth 0 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 10 monclient: _check_auth_tickets 2024-01-27T04:20:24.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 1 -- 172.21.15.110:0/3493593524 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44280315a0 con 0x7f442c06bcb0 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 1 -- 172.21.15.110:0/3493593524 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428031930 con 0x7f442c06bcb0 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f443303f640 5 monclient: authenticate success, global_id 18911 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f443303f640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f443303f640 4 set_mon_vals no callback set 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.384+0000 7f44237fe640 10 monclient: dump: 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.389 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 10 monclient: shutdown 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 1 -- 172.21.15.110:0/3493593524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c06bcb0 msgr2=0x7f442c0ae4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 1 --2- 172.21.15.110:0/3493593524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c06bcb0 0x7f442c0ae4b0 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7f442c0672e0 tx=0x7f442802b840 comp rx=0 tx=0).stop 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 1 -- 172.21.15.110:0/3493593524 shutdown_connections 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 1 --2- 172.21.15.110:0/3493593524 >> v2:172.21.15.110:3300/0 conn(0x7f442c142c10 0x7f442c13f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 1 --2- 172.21.15.110:0/3493593524 >> v2:172.21.15.162:3300/0 conn(0x7f442c06c090 0x7f442c1426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 1 --2- 172.21.15.110:0/3493593524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c06bcb0 0x7f442c0ae4b0 unknown :-1 s=CLOSED pgs=3415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 1 -- 172.21.15.110:0/3493593524 >> 172.21.15.110:0/3493593524 conn(0x7f442c0b5b40 msgr2=0x7f442c0b5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 1 -- 172.21.15.110:0/3493593524 shutdown_connections 2024-01-27T04:20:24.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.385+0000 7f443303f640 1 -- 172.21.15.110:0/3493593524 wait complete. 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.386+0000 7f443303f640 10 monclient: build_initial_monmap 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.386+0000 7f443303f640 10 monclient: monmap: 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.386+0000 7f443303f640 1 librados: starting msgr at 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.386+0000 7f443303f640 1 librados: starting objecter 2024-01-27T04:20:24.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.386+0000 7f443303f640 1 Processor -- start 2024-01-27T04:20:24.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.386+0000 7f443303f640 1 -- start start 2024-01-27T04:20:24.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.386+0000 7f443303f640 1 librados: setting wanted keys 2024-01-27T04:20:24.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.386+0000 7f443303f640 1 librados: calling monclient init 2024-01-27T04:20:24.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.386+0000 7f443303f640 10 monclient: init 2024-01-27T04:20:24.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:24.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T04:20:24.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f442c06bcb0 0x7f442c0b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient(hunting): picked mon.noname-a con 0x7f442c06bcb0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:24.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f442c06c090 0x7f442c0b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient(hunting): picked mon.noname-b con 0x7f442c06c090 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c142c10 0x7f442c140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient(hunting): picked mon.noname-c con 0x7f442c142c10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c0b4e20 con 0x7f442c06bcb0 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c0b50d0 con 0x7f442c06c090 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442c0b3730 con 0x7f442c142c10 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f443303f640 10 monclient(hunting): authenticate will time out at 3707.741396s 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f44315dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c142c10 0x7f442c140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.387+0000 7f4430ddb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f442c06bcb0 0x7f442c0b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4423fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f442c06c090 0x7f442c0b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f44315dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c142c10 0x7f442c140170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40778/0 (socket says 172.21.15.110:40778) 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4430ddb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f442c06bcb0 0x7f442c0b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44964/0 (socket says 172.21.15.110:44964) 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f44315dc640 1 -- 172.21.15.110:0/18990214 learned_addr learned my addr 172.21.15.110:0/18990214 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f44315dc640 10 monclient(hunting): get_auth_request con 0x7f442c142c10 auth_method 0 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f44315dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f44315dc640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f44315dc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4423fff640 10 monclient(hunting): get_auth_request con 0x7f442c06c090 auth_method 0 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4423fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4423fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4423fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4430ddb640 10 monclient(hunting): get_auth_request con 0x7f442c06bcb0 auth_method 0 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4430ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4430ddb640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4430ddb640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4423fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4423fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4423fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f44315dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f44315dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f44315dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4430ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4430ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.388+0000 7f4430ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 10 monclient(hunting): handle_auth_done global_id 18914 payload 824 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 1 -- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f442c06c090 msgr2=0x7f442c0b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f442c06c090 0x7f442c0b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 1 -- 172.21.15.110:0/18990214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f442c06bcb0 msgr2=0x7f442c0b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f442c06bcb0 0x7f442c0b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 10 monclient: _finish_hunting 0 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 1 monclient: found mon.noname-c 2024-01-27T04:20:24.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:24.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T04:20:24.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.391+0000 7f44315dc640 1 -- 172.21.15.110:0/18990214 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c009400 con 0x7f442c142c10 2024-01-27T04:20:24.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.393+0000 7f44315dc640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c142c10 0x7f442c140170 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 crypto rx=0x7f442400efc0 tx=0x7f442400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:24.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.393+0000 7f44217fa640 10 client.?.objecter ms_handle_connect 0x7f442c142c10 2024-01-27T04:20:24.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.393+0000 7f44217fa640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 1 -- 172.21.15.110:0/18990214 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424018070 con 0x7f442c142c10 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 10 monclient: dump: 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.399 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 10 monclient: _finish_auth 0 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 10 monclient: _check_auth_tickets 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 1 -- 172.21.15.110:0/18990214 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442400f040 con 0x7f442c142c10 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 1 -- 172.21.15.110:0/18990214 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442400ce00 con 0x7f442c142c10 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f44217fa640 10 monclient: dump: 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:24.400 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 5 monclient: authenticate success, global_id 18914 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 10 monclient: _renew_subs 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4424009b30 con 0x7f442c142c10 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f4422ffd640 4 set_mon_vals no callback set 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 10 client.18914.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:24.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 10 monclient: _renew_subs 2024-01-27T04:20:24.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:24.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4428009400 con 0x7f442c142c10 2024-01-27T04:20:24.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 1 librados: init done 2024-01-27T04:20:24.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 10 client.18914.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:24.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 15 client.18914.objecter _session_command_op_assign -1 1 2024-01-27T04:20:24.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 20 client.18914.objecter _get_session osd=-1 returning homeless 2024-01-27T04:20:24.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 10 client.18914.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:24.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:24.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.397+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f442c145510 con 0x7f442c142c10 2024-01-27T04:20:24.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.416+0000 7f44217fa640 1 -- 172.21.15.110:0/18990214 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f4424014480 con 0x7f442c142c10 2024-01-27T04:20:24.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.416+0000 7f44217fa640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f441806c9f0 0x7f441806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.417+0000 7f4430ddb640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f441806c9f0 0x7f441806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.417+0000 7f4430ddb640 10 monclient: get_auth_request con 0x7f441806c9f0 auth_method 0 2024-01-27T04:20:24.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.417+0000 7f4430ddb640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f441806c9f0 0x7f441806eeb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4428036000 tx=0x7f442803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:24.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.417+0000 7f44217fa640 10 client.18914.objecter ms_handle_connect 0x7f441806c9f0 2024-01-27T04:20:24.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.418+0000 7f44217fa640 1 -- 172.21.15.110:0/18990214 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f4424059200 con 0x7f442c142c10 2024-01-27T04:20:24.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.418+0000 7f44217fa640 10 client.18914.objecter ms_dispatch 0x7f442c000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:24.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.418+0000 7f44217fa640 3 client.18914.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:24.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.418+0000 7f44217fa640 3 client.18914.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:24.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 10 client.18914.objecter checking command 1 2024-01-27T04:20:24.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7f4418074770 0x7f4418076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 20 client.18914.objecter _get_session s=0x7f44180740a0 osd=4 3 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 20 client.18914.objecter put_session s=0x7f44180740a0 osd=4 3 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 15 client.18914.objecter _session_command_op_remove -1 1 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 20 client.18914.objecter _get_session s=0x7f44180740a0 osd=4 3 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 20 client.18914.objecter get_session s=0x7f44180740a0 osd=4 3 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 15 client.18914.objecter _session_command_op_assign 4 1 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 20 client.18914.objecter put_session s=0x7f44180740a0 osd=4 4 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 10 client.18914.objecter _send_command 1 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 1 -- 172.21.15.110:0/18990214 --> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44180772e0 con 0x7f4418074770 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 20 client.18914.objecter dump_active .. 0 homeless 2024-01-27T04:20:24.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f4423fff640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7f4418074770 0x7f4418076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 1 -- 172.21.15.110:0/18990214 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=1334) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44240554a0 con 0x7f442c142c10 2024-01-27T04:20:24.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.419+0000 7f44217fa640 10 monclient: handle_get_version_reply finishing 1 version 1334 2024-01-27T04:20:24.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.420+0000 7f4423fff640 10 monclient: get_auth_request con 0x7f4418074770 auth_method 0 2024-01-27T04:20:24.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.422+0000 7f4423fff640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7f4418074770 0x7f4418076c10 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:24.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.422+0000 7f44217fa640 10 client.18914.objecter ms_handle_connect 0x7f4418074770 2024-01-27T04:20:24.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.446+0000 7f44217fa640 1 -- 172.21.15.110:0/18990214 <== osd.4 v2:172.21.15.162:6826/1007517023 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f44180772e0 con 0x7f4418074770 2024-01-27T04:20:24.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.446+0000 7f44217fa640 10 client.18914.objecter ms_dispatch 0x7f442c000b90 command_reply(tid 1: 0 ) v1 2024-01-27T04:20:24.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.446+0000 7f44217fa640 10 client.18914.objecter _finish_command 1 = system:0 2024-01-27T04:20:24.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.446+0000 7f44217fa640 20 client.18914.objecter put_session s=0x7f44180740a0 osd=4 4 2024-01-27T04:20:24.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.446+0000 7f44217fa640 15 client.18914.objecter _session_command_op_remove 4 1 2024-01-27T04:20:24.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 10 client.18914.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-27T04:20:24.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 15 client.18914.objecter _session_command_op_assign -1 2 2024-01-27T04:20:24.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 20 client.18914.objecter _get_session s=0x7f44180740a0 osd=4 3 2024-01-27T04:20:24.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 20 client.18914.objecter put_session s=0x7f44180740a0 osd=4 3 2024-01-27T04:20:24.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 20 client.18914.objecter _get_session s=0x7f44180740a0 osd=4 3 2024-01-27T04:20:24.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 15 client.18914.objecter _session_command_op_remove -1 2 2024-01-27T04:20:24.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 20 client.18914.objecter get_session s=0x7f44180740a0 osd=4 3 2024-01-27T04:20:24.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 15 client.18914.objecter _session_command_op_assign 4 2 2024-01-27T04:20:24.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 20 client.18914.objecter put_session s=0x7f44180740a0 osd=4 4 2024-01-27T04:20:24.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 10 client.18914.objecter _send_command 2 2024-01-27T04:20:24.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.459+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 --> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f442c067db0 con 0x7f4418074770 2024-01-27T04:20:24.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.474+0000 7f44217fa640 1 -- 172.21.15.110:0/18990214 <== osd.4 v2:172.21.15.162:6826/1007517023 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f442c067db0 con 0x7f4418074770 2024-01-27T04:20:24.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.474+0000 7f44217fa640 10 client.18914.objecter ms_dispatch 0x7f442c000b90 command_reply(tid 2: 0 ) v1 2024-01-27T04:20:24.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.474+0000 7f44217fa640 10 client.18914.objecter _finish_command 2 = system:0 2024-01-27T04:20:24.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.474+0000 7f44217fa640 20 client.18914.objecter put_session s=0x7f44180740a0 osd=4 4 2024-01-27T04:20:24.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.474+0000 7f44217fa640 15 client.18914.objecter _session_command_op_remove 4 2 2024-01-27T04:20:24.476 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-01-27T04:20:24.476 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-27T04:20:24.476 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 10 librados: watch_flush enter 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 10 librados: watch_flush exit 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 10 client.18914.objecter close_session for osd.4 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7f4418074770 msgr2=0x7f4418076c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6826/1007517023,v1:172.21.15.162:6827/1007517023] conn(0x7f4418074770 0x7f4418076c10 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 20 client.18914.objecter put_session s=0x7f44180740a0 osd=4 1 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 20 client.18914.objecter shutdown clearing up homeless session... 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 10 client.18914.objecter successfully canceled tick 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f441806c9f0 msgr2=0x7f441806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f441806c9f0 0x7f441806eeb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4428036000 tx=0x7f442803d040 comp rx=0 tx=0).stop 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 10 monclient: shutdown 2024-01-27T04:20:24.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:24.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c142c10 msgr2=0x7f442c140170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 --2- 172.21.15.110:0/18990214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f442c142c10 0x7f442c140170 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 crypto rx=0x7f442400efc0 tx=0x7f442400c490 comp rx=0 tx=0).stop 2024-01-27T04:20:24.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f44315dc640 1 -- 172.21.15.110:0/18990214 reap_dead start 2024-01-27T04:20:24.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 shutdown_connections 2024-01-27T04:20:24.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 >> 172.21.15.110:0/18990214 conn(0x7f442c0b5b40 msgr2=0x7f442c0b5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:24.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 shutdown_connections 2024-01-27T04:20:24.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 -- 172.21.15.110:0/18990214 wait complete. 2024-01-27T04:20:24.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.475+0000 7f443303f640 1 librados: shutdown 2024-01-27T04:20:24.487 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-27T04:20:24.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.586+0000 7f2b0d11d640 10 monclient: get_monmap_and_config 2024-01-27T04:20:24.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.586+0000 7f2b0d11d640 10 monclient: build_initial_monmap 2024-01-27T04:20:24.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.586+0000 7f2b0d11d640 10 monclient: monmap: 2024-01-27T04:20:24.587 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:24.587 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:24.587463+0000 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:24.587463+0000 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:24.588 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.586+0000 7f2b0d11d640 1 Processor -- start 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.586+0000 7f2b0d11d640 1 -- start start 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.586+0000 7f2b0d11d640 10 monclient: init 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2b08066700 0x7f2b08071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 10 monclient(hunting): picked mon.noname-c con 0x7f2b08066700 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b080adad0 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b08143b60 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:24.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0813a380 con 0x7f2b08143b60 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b08066ae0 con 0x7f2b080adad0 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b08072890 con 0x7f2b08066700 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.587+0000 7f2b0d11d640 10 monclient(hunting): authenticate will time out at 3707.941558s 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.588+0000 7f2b0759e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2b08143b60 0x7f2b0814a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48094/0 (socket says 172.21.15.110:48094) 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.588+0000 7f2b0759e640 1 -- 172.21.15.110:0/3303068122 learned_addr learned my addr 172.21.15.110:0/3303068122 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.588+0000 7f2b06d9d640 1 --2- 172.21.15.110:0/3303068122 >> v2:172.21.15.162:3300/0 conn(0x7f2b08066700 0x7f2b08071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.588+0000 7f2b06d9d640 10 monclient(hunting): get_auth_request con 0x7f2b08066700 auth_method 0 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.588+0000 7f2b06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.588+0000 7f2b06d9d640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.588+0000 7f2b06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.589+0000 7f2b06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.589+0000 7f2b06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.589+0000 7f2b06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.590+0000 7f2b05d9b640 1 -- 172.21.15.110:0/3303068122 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690961818 0 0) 0x7f2af8001110 con 0x7f2b08143b60 2024-01-27T04:20:24.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.590+0000 7f2b05d9b640 10 monclient(hunting): _negotiate 2024-01-27T04:20:24.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.590+0000 7f2b05d9b640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.590+0000 7f2b05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.590+0000 7f2b05d9b640 10 monclient(hunting): my global_id is 17046 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.590+0000 7f2b05d9b640 1 -- 172.21.15.110:0/3303068122 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af0003520 con 0x7f2b08143b60 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.591+0000 7f2b06d9d640 10 monclient(hunting): handle_auth_done global_id 20956 payload 274 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.591+0000 7f2b06d9d640 1 -- 172.21.15.110:0/3303068122 >> v1:172.21.15.162:6789/0 conn(0x7f2b080adad0 legacy=0x7f2b08143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.591+0000 7f2b06d9d640 1 -- 172.21.15.110:0/3303068122 >> v1:172.21.15.110:6789/0 conn(0x7f2b08143b60 legacy=0x7f2b0814a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.591+0000 7f2b06d9d640 10 monclient: _finish_hunting 0 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.591+0000 7f2b06d9d640 1 monclient: found mon.noname-c 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.591+0000 7f2b06d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.591+0000 7f2b06d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T04:20:24.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.591+0000 7f2b06d9d640 1 -- 172.21.15.110:0/3303068122 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b08070010 con 0x7f2b08066700 2024-01-27T04:20:24.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.595+0000 7f2b06d9d640 1 --2- 172.21.15.110:0/3303068122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b08066700 0x7f2b08071300 secure :-1 s=READY pgs=5096 cs=0 l=1 rev1=1 crypto rx=0x7f2b000097a0 tx=0x7f2b0002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2febe1558e7e97ab server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:24.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 1 -- 172.21.15.110:0/3303068122 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b00031830 con 0x7f2b08066700 2024-01-27T04:20:24.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 10 monclient: dump: 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 10 monclient: _finish_auth 0 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 10 monclient: _check_auth_tickets 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 1 -- 172.21.15.110:0/3303068122 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b00031990 con 0x7f2b08066700 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b0d11d640 5 monclient: authenticate success, global_id 20956 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b0d11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b0d11d640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 1 -- 172.21.15.110:0/3303068122 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b00031d20 con 0x7f2b08066700 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b0d11d640 4 set_mon_vals no callback set 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.599+0000 7f2b05d9b640 10 monclient: dump: 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.602 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.603 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.603 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:24.603 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.603 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:24.603 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:24.603 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:24.603 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.600+0000 7f2b0d11d640 10 monclient: shutdown 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.600+0000 7f2b0d11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.600+0000 7f2b0d11d640 1 -- 172.21.15.110:0/3303068122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b08066700 msgr2=0x7f2b08071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.600+0000 7f2b0d11d640 1 --2- 172.21.15.110:0/3303068122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b08066700 0x7f2b08071300 secure :-1 s=READY pgs=5096 cs=0 l=1 rev1=1 crypto rx=0x7f2b000097a0 tx=0x7f2b0002b8d0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.600+0000 7f2b0d11d640 1 -- 172.21.15.110:0/3303068122 shutdown_connections 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.600+0000 7f2b0d11d640 1 --2- 172.21.15.110:0/3303068122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b08066700 0x7f2b08071300 unknown :-1 s=CLOSED pgs=5096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.600+0000 7f2b0d11d640 1 -- 172.21.15.110:0/3303068122 >> 172.21.15.110:0/3303068122 conn(0x7f2b080b4620 msgr2=0x7f2b080b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.600+0000 7f2b0d11d640 1 -- 172.21.15.110:0/3303068122 shutdown_connections 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.600+0000 7f2b0d11d640 1 -- 172.21.15.110:0/3303068122 wait complete. 2024-01-27T04:20:24.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.601+0000 7f2b0d11d640 10 monclient: build_initial_monmap 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.601+0000 7f2b0d11d640 10 monclient: monmap: 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.601+0000 7f2b0d11d640 1 librados: starting msgr at 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.601+0000 7f2b0d11d640 1 librados: starting objecter 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.601+0000 7f2b0d11d640 1 Processor -- start 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 1 -- start start 2024-01-27T04:20:24.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 1 librados: setting wanted keys 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 1 librados: calling monclient init 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient: init 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b08066700 0x7f2b0813c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient(hunting): picked mon.noname-c con 0x7f2b08066700 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b080adad0 0x7f2b081daaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient(hunting): picked mon.noname-b con 0x7f2b080adad0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b08143b60 0x7f2b0806f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient(hunting): picked mon.noname-a con 0x7f2b08143b60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0813dcd0 con 0x7f2b08143b60 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0813f660 con 0x7f2b080adad0 2024-01-27T04:20:24.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0813f8d0 con 0x7f2b08066700 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.602+0000 7f2b0d11d640 10 monclient(hunting): authenticate will time out at 3707.956510s 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b06d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b08066700 0x7f2b0813c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b080adad0 0x7f2b081daaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b080adad0 0x7f2b081daaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:58924/0 (socket says 172.21.15.110:58924) 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 1 -- 172.21.15.110:0/476469697 learned_addr learned my addr 172.21.15.110:0/476469697 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 10 monclient(hunting): get_auth_request con 0x7f2b080adad0 auth_method 0 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b06d9d640 10 monclient(hunting): get_auth_request con 0x7f2b08066700 auth_method 0 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b06d9d640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.603+0000 7f2b0659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.604+0000 7f2b06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.604+0000 7f2b06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.604+0000 7f2b06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.605+0000 7f2b0759e640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b08143b60 0x7f2b0806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.605+0000 7f2b0759e640 10 monclient(hunting): get_auth_request con 0x7f2b08143b60 auth_method 0 2024-01-27T04:20:24.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.605+0000 7f2b0759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:24.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.605+0000 7f2b0759e640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.605+0000 7f2b0759e640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 10 monclient(hunting): handle_auth_done global_id 20962 payload 824 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 1 -- 172.21.15.110:0/476469697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b08066700 msgr2=0x7f2b0813c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b08066700 0x7f2b0813c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 1 -- 172.21.15.110:0/476469697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b08143b60 msgr2=0x7f2b0806f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b08143b60 0x7f2b0806f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 10 monclient: _finish_hunting 0 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 1 monclient: found mon.noname-b 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:24.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:24.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.606+0000 7f2b0659c640 1 -- 172.21.15.110:0/476469697 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b00009400 con 0x7f2b080adad0 2024-01-27T04:20:24.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.608+0000 7f2b0659c640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b080adad0 0x7f2b081daaf0 secure :-1 s=READY pgs=5097 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00c870 tx=0x7f2afc00cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:24.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.608+0000 7f2aef7fe640 10 client.?.objecter ms_handle_connect 0x7f2b080adad0 2024-01-27T04:20:24.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.608+0000 7f2aef7fe640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 1 -- 172.21.15.110:0/476469697 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc003b70 con 0x7f2b080adad0 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 10 monclient: dump: 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.614 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 10 monclient: _finish_auth 0 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 10 monclient: _check_auth_tickets 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 1 -- 172.21.15.110:0/476469697 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2afc003cf0 con 0x7f2b080adad0 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0d11d640 5 monclient: authenticate success, global_id 20962 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 1 -- 172.21.15.110:0/476469697 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc00aa90 con 0x7f2b080adad0 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0d11d640 10 monclient: _renew_subs 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0d11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0806fc70 con 0x7f2b080adad0 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:24.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2aef7fe640 10 monclient: dump: 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0559a640 4 set_mon_vals no callback set 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0d11d640 10 client.20962.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:24.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0d11d640 10 monclient: _renew_subs 2024-01-27T04:20:24.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0d11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:24.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0813e340 con 0x7f2b080adad0 2024-01-27T04:20:24.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.612+0000 7f2b0d11d640 1 librados: init done 2024-01-27T04:20:24.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.613+0000 7f2b0d11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:24.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.613+0000 7f2b0d11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:24.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.613+0000 7f2b0d11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:24.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.613+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0813fdf0 con 0x7f2b080adad0 2024-01-27T04:20:24.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.857+0000 7f2aef7fe640 1 -- 172.21.15.110:0/476469697 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f2afc00abf0 con 0x7f2b080adad0 2024-01-27T04:20:24.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.857+0000 7f2aef7fe640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f2af406c9a0 0x7f2af406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:24.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.858+0000 7f2b06d9d640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f2af406c9a0 0x7f2af406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:24.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.858+0000 7f2b06d9d640 10 monclient: get_auth_request con 0x7f2af406c9a0 auth_method 0 2024-01-27T04:20:24.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.858+0000 7f2b06d9d640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f2af406c9a0 0x7f2af406ee60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2b00036000 tx=0x7f2b0003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:24.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.858+0000 7f2aef7fe640 10 client.20962.objecter ms_handle_connect 0x7f2af406c9a0 2024-01-27T04:20:24.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.862+0000 7f2aef7fe640 1 -- 172.21.15.110:0/476469697 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f2afc058a80 con 0x7f2b080adad0 2024-01-27T04:20:24.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.862+0000 7f2aef7fe640 10 client.20962.objecter ms_dispatch 0x7f2b08000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:24.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.862+0000 7f2aef7fe640 3 client.20962.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:24.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.862+0000 7f2aef7fe640 3 client.20962.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:24.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.862+0000 7f2aef7fe640 20 client.20962.objecter dump_active .. 0 homeless 2024-01-27T04:20:24.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.872+0000 7f2aef7fe640 1 -- 172.21.15.110:0/476469697 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f2afc054d20 con 0x7f2b080adad0 2024-01-27T04:20:24.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.872+0000 7f2aef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:24.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:24.872+0000 7f2aef7fe640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:25.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.057+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b080ae870 con 0x7f2af406c9a0 2024-01-27T04:20:25.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.058+0000 7f2aef7fe640 1 -- 172.21.15.110:0/476469697 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2b00031b80 con 0x7f2af406c9a0 2024-01-27T04:20:25.059 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 4 to deep-scrub 2024-01-27T04:20:25.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 10 librados: watch_flush enter 2024-01-27T04:20:25.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 10 librados: watch_flush exit 2024-01-27T04:20:25.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 20 client.20962.objecter shutdown clearing up homeless session... 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 10 client.20962.objecter successfully canceled tick 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f2af406c9a0 msgr2=0x7f2af406ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f2af406c9a0 0x7f2af406ee60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2b00036000 tx=0x7f2b0003c040 comp rx=0 tx=0).stop 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 10 monclient: shutdown 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b080adad0 msgr2=0x7f2b081daaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b080adad0 0x7f2b081daaf0 secure :-1 s=READY pgs=5097 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00c870 tx=0x7f2afc00cd30 comp rx=0 tx=0).stop 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 shutdown_connections 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f2af406c9a0 0x7f2af406ee60 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b08143b60 0x7f2b0806f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b080adad0 0x7f2b081daaf0 unknown :-1 s=CLOSED pgs=5097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 --2- 172.21.15.110:0/476469697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b08066700 0x7f2b0813c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 >> 172.21.15.110:0/476469697 conn(0x7f2b080b4620 msgr2=0x7f2b080b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:25.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 shutdown_connections 2024-01-27T04:20:25.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 -- 172.21.15.110:0/476469697 wait complete. 2024-01-27T04:20:25.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.060+0000 7f2b0d11d640 1 librados: shutdown 2024-01-27T04:20:25.073 INFO:tasks.ceph:Scrubbing osd.5 2024-01-27T04:20:25.074 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-27T04:20:25.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.175+0000 7f447ad7d640 10 monclient: get_monmap_and_config 2024-01-27T04:20:25.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.175+0000 7f447ad7d640 10 monclient: build_initial_monmap 2024-01-27T04:20:25.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.175+0000 7f447ad7d640 10 monclient: monmap: 2024-01-27T04:20:25.177 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:25.177 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.177 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:25.176218+0000 2024-01-27T04:20:25.177 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:25.176218+0000 2024-01-27T04:20:25.177 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:25.177 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.177 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:25.178 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:25.178 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:25.178 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:25.178 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:25.178 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.175+0000 7f447ad7d640 1 Processor -- start 2024-01-27T04:20:25.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.175+0000 7f447ad7d640 1 -- start start 2024-01-27T04:20:25.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.175+0000 7f447ad7d640 10 monclient: init 2024-01-27T04:20:25.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:25.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-01-27T04:20:25.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f447406a940 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4474142de0 0x7f44740b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 10 monclient(hunting): picked mon.noname-a con 0x7f4474142de0 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44740a9e40 0x7f44740b3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 10 monclient(hunting): picked mon.noname-b con 0x7f44740a9e40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44740b37c0 con 0x7f447406a940 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447406ad20 con 0x7f4474142de0 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44740aac50 con 0x7f44740a9e40 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.176+0000 7f447ad7d640 10 monclient(hunting): authenticate will time out at 3708.530455s 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f4473fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4474142de0 0x7f44740b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f447931a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44740a9e40 0x7f44740b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f4478b19640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f447406a940 0x7f44741426d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:34632/0 (socket says 172.21.15.110:34632) 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f4478b19640 1 -- 172.21.15.110:0/51949121 learned_addr learned my addr 172.21.15.110:0/51949121 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T04:20:25.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f447931a640 10 monclient(hunting): get_auth_request con 0x7f44740a9e40 auth_method 0 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f447931a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f447931a640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f447931a640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f4473fff640 10 monclient(hunting): get_auth_request con 0x7f4474142de0 auth_method 0 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f4473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f4473fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.177+0000 7f4473fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.178+0000 7f447931a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.178+0000 7f447931a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.178+0000 7f447931a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.178+0000 7f4473fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.178+0000 7f4473fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.178+0000 7f4473fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.179+0000 7f44737fe640 1 -- 172.21.15.110:0/51949121 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775216454 0 0) 0x7f44740b37c0 con 0x7f447406a940 2024-01-27T04:20:25.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.179+0000 7f44737fe640 10 monclient(hunting): _negotiate 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.179+0000 7f44737fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.179+0000 7f44737fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.179+0000 7f44737fe640 10 monclient(hunting): my global_id is 20965 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.179+0000 7f44737fe640 1 -- 172.21.15.110:0/51949121 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f445c003520 con 0x7f447406a940 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f447931a640 10 monclient(hunting): handle_auth_done global_id 18920 payload 274 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f447931a640 1 -- 172.21.15.110:0/51949121 >> v2:172.21.15.110:3300/0 conn(0x7f4474142de0 msgr2=0x7f44740b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f447931a640 1 --2- 172.21.15.110:0/51949121 >> v2:172.21.15.110:3300/0 conn(0x7f4474142de0 0x7f44740b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f447931a640 1 -- 172.21.15.110:0/51949121 >> v1:172.21.15.162:6789/0 conn(0x7f447406a940 legacy=0x7f44741426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f447931a640 10 monclient: _finish_hunting 0 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f447931a640 1 monclient: found mon.noname-b 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f447931a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f447931a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f447931a640 1 -- 172.21.15.110:0/51949121 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44740aaf30 con 0x7f44740a9e40 2024-01-27T04:20:25.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.180+0000 7f4473fff640 1 --2- 172.21.15.110:0/51949121 >> v2:172.21.15.110:3300/0 conn(0x7f4474142de0 0x7f44740b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T04:20:25.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.183+0000 7f447931a640 1 --2- 172.21.15.110:0/51949121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44740a9e40 0x7f44740b3280 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7f446c009d90 tx=0x7f446c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=418bfa9793b34020 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:25.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 1 -- 172.21.15.110:0/51949121 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c02cc50 con 0x7f44740a9e40 2024-01-27T04:20:25.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 10 monclient: dump: 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:25.188 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 10 monclient: _finish_auth 0 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 10 monclient: _check_auth_tickets 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 1 -- 172.21.15.110:0/51949121 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446c02cdb0 con 0x7f44740a9e40 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f447ad7d640 5 monclient: authenticate success, global_id 18920 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 1 -- 172.21.15.110:0/51949121 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c034660 con 0x7f44740a9e40 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f447ad7d640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f447ad7d640 4 set_mon_vals no callback set 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:25.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.186+0000 7f44737fe640 10 monclient: dump: 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:25.190 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 10 monclient: shutdown 2024-01-27T04:20:25.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:25.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 1 -- 172.21.15.110:0/51949121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44740a9e40 msgr2=0x7f44740b3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 1 --2- 172.21.15.110:0/51949121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44740a9e40 0x7f44740b3280 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7f446c009d90 tx=0x7f446c02bea0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 1 -- 172.21.15.110:0/51949121 shutdown_connections 2024-01-27T04:20:25.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 1 --2- 172.21.15.110:0/51949121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44740a9e40 0x7f44740b3280 unknown :-1 s=CLOSED pgs=3417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 1 --2- 172.21.15.110:0/51949121 >> v2:172.21.15.110:3300/0 conn(0x7f4474142de0 0x7f44740b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 1 -- 172.21.15.110:0/51949121 >> 172.21.15.110:0/51949121 conn(0x7f44740b7d70 msgr2=0x7f44740b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 1 -- 172.21.15.110:0/51949121 shutdown_connections 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.187+0000 7f447ad7d640 1 -- 172.21.15.110:0/51949121 wait complete. 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.188+0000 7f447ad7d640 10 monclient: build_initial_monmap 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.188+0000 7f447ad7d640 10 monclient: monmap: 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:25.194 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.188+0000 7f447ad7d640 1 librados: starting msgr at 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.188+0000 7f447ad7d640 1 librados: starting objecter 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.188+0000 7f447ad7d640 1 Processor -- start 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.188+0000 7f447ad7d640 1 -- start start 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.188+0000 7f447ad7d640 1 librados: setting wanted keys 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.188+0000 7f447ad7d640 1 librados: calling monclient init 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.188+0000 7f447ad7d640 10 monclient: init 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f447406a940 0x7f44740adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient(hunting): picked mon.noname-c con 0x7f447406a940 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44740a9e40 0x7f44740b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient(hunting): picked mon.noname-a con 0x7f44740a9e40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:25.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4474142de0 0x7f44740b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient(hunting): picked mon.noname-b con 0x7f4474142de0 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447413ac90 con 0x7f44740a9e40 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44740b1010 con 0x7f4474142de0 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44740afb40 con 0x7f447406a940 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f447ad7d640 10 monclient(hunting): authenticate will time out at 3708.543102s 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f4478b19640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f447406a940 0x7f44740adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f4473fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44740a9e40 0x7f44740b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f4478b19640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f447406a940 0x7f44740adca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40818/0 (socket says 172.21.15.110:40818) 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f4478b19640 1 -- 172.21.15.110:0/3195637301 learned_addr learned my addr 172.21.15.110:0/3195637301 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f4478b19640 10 monclient(hunting): get_auth_request con 0x7f447406a940 auth_method 0 2024-01-27T04:20:25.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f4478b19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f4478b19640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.189+0000 7f4478b19640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f447931a640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4474142de0 0x7f44740b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4473fff640 10 monclient(hunting): get_auth_request con 0x7f44740a9e40 auth_method 0 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4473fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4473fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f447931a640 10 monclient(hunting): get_auth_request con 0x7f4474142de0 auth_method 0 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f447931a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f447931a640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f447931a640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4478b19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4478b19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4478b19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4473fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4473fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f4473fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f447931a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f447931a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.190+0000 7f447931a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 10 monclient(hunting): handle_auth_done global_id 18923 payload 824 2024-01-27T04:20:25.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 1 -- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4474142de0 msgr2=0x7f44740b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4474142de0 0x7f44740b0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 1 -- 172.21.15.110:0/3195637301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44740a9e40 msgr2=0x7f44740b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44740a9e40 0x7f44740b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 10 monclient: _finish_hunting 0 2024-01-27T04:20:25.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 1 monclient: found mon.noname-c 2024-01-27T04:20:25.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:25.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.193+0000 7f4478b19640 1 -- 172.21.15.110:0/3195637301 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4468009400 con 0x7f447406a940 2024-01-27T04:20:25.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.195+0000 7f4478b19640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f447406a940 0x7f44740adca0 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7f446400c440 tx=0x7f446400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:25.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.195+0000 7f44717fa640 10 client.?.objecter ms_handle_connect 0x7f447406a940 2024-01-27T04:20:25.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.195+0000 7f44717fa640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:25.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 1 -- 172.21.15.110:0/3195637301 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4464005ce0 con 0x7f447406a940 2024-01-27T04:20:25.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 10 monclient: dump: 2024-01-27T04:20:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 10 monclient: _finish_auth 0 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 10 monclient: _check_auth_tickets 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 1 -- 172.21.15.110:0/3195637301 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4464005e40 con 0x7f447406a940 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 1 -- 172.21.15.110:0/3195637301 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44640072b0 con 0x7f447406a940 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f447ad7d640 5 monclient: authenticate success, global_id 18923 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f447ad7d640 10 monclient: _renew_subs 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f447ad7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c009a70 con 0x7f447406a940 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f44717fa640 10 monclient: dump: 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:25.204 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f4472ffd640 4 set_mon_vals no callback set 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f447ad7d640 10 client.18923.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f447ad7d640 10 monclient: _renew_subs 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f447ad7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44740ae860 con 0x7f447406a940 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.198+0000 7f447ad7d640 1 librados: init done 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.199+0000 7f447ad7d640 10 client.18923.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.199+0000 7f447ad7d640 15 client.18923.objecter _session_command_op_assign -1 1 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.199+0000 7f447ad7d640 20 client.18923.objecter _get_session osd=-1 returning homeless 2024-01-27T04:20:25.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.199+0000 7f447ad7d640 10 client.18923.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:25.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.199+0000 7f447ad7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.199+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f44740b4590 con 0x7f447406a940 2024-01-27T04:20:25.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.218+0000 7f44717fa640 1 -- 172.21.15.110:0/3195637301 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f4464007410 con 0x7f447406a940 2024-01-27T04:20:25.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.218+0000 7f44717fa640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f446006c9f0 0x7f446006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.219+0000 7f4473fff640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f446006c9f0 0x7f446006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.219+0000 7f4473fff640 10 monclient: get_auth_request con 0x7f446006c9f0 auth_method 0 2024-01-27T04:20:25.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.219+0000 7f4473fff640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f446006c9f0 0x7f446006eeb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f44740b4560 tx=0x7f4468009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:25.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.219+0000 7f44717fa640 10 client.18923.objecter ms_handle_connect 0x7f446006c9f0 2024-01-27T04:20:25.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 1 -- 172.21.15.110:0/3195637301 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f4464058070 con 0x7f447406a940 2024-01-27T04:20:25.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 10 client.18923.objecter ms_dispatch 0x7f4474000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:25.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 3 client.18923.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:25.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 3 client.18923.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:25.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 10 client.18923.objecter checking command 1 2024-01-27T04:20:25.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f4460074770 0x7f4460076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 20 client.18923.objecter _get_session s=0x7f44600740a0 osd=5 3 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 20 client.18923.objecter put_session s=0x7f44600740a0 osd=5 3 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 15 client.18923.objecter _session_command_op_remove -1 1 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 20 client.18923.objecter _get_session s=0x7f44600740a0 osd=5 3 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 20 client.18923.objecter get_session s=0x7f44600740a0 osd=5 3 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 15 client.18923.objecter _session_command_op_assign 5 1 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 20 client.18923.objecter put_session s=0x7f44600740a0 osd=5 4 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 10 client.18923.objecter _send_command 1 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 1 -- 172.21.15.110:0/3195637301 --> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44600772e0 con 0x7f4460074770 2024-01-27T04:20:25.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 20 client.18923.objecter dump_active .. 0 homeless 2024-01-27T04:20:25.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f447931a640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f4460074770 0x7f4460076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 1 -- 172.21.15.110:0/3195637301 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=1334) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4464057e40 con 0x7f447406a940 2024-01-27T04:20:25.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.221+0000 7f44717fa640 10 monclient: handle_get_version_reply finishing 1 version 1334 2024-01-27T04:20:25.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.222+0000 7f447931a640 10 monclient: get_auth_request con 0x7f4460074770 auth_method 0 2024-01-27T04:20:25.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.224+0000 7f447931a640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f4460074770 0x7f4460076c10 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:25.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.225+0000 7f44717fa640 10 client.18923.objecter ms_handle_connect 0x7f4460074770 2024-01-27T04:20:25.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.248+0000 7f44717fa640 1 -- 172.21.15.110:0/3195637301 <== osd.5 v2:172.21.15.162:6802/946981202 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f44600772e0 con 0x7f4460074770 2024-01-27T04:20:25.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.248+0000 7f44717fa640 10 client.18923.objecter ms_dispatch 0x7f4474000b90 command_reply(tid 1: 0 ) v1 2024-01-27T04:20:25.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.248+0000 7f44717fa640 10 client.18923.objecter _finish_command 1 = system:0 2024-01-27T04:20:25.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.248+0000 7f44717fa640 20 client.18923.objecter put_session s=0x7f44600740a0 osd=5 4 2024-01-27T04:20:25.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.248+0000 7f44717fa640 15 client.18923.objecter _session_command_op_remove 5 1 2024-01-27T04:20:25.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 10 client.18923.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-27T04:20:25.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 15 client.18923.objecter _session_command_op_assign -1 2 2024-01-27T04:20:25.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 20 client.18923.objecter _get_session s=0x7f44600740a0 osd=5 3 2024-01-27T04:20:25.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 20 client.18923.objecter put_session s=0x7f44600740a0 osd=5 3 2024-01-27T04:20:25.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 20 client.18923.objecter _get_session s=0x7f44600740a0 osd=5 3 2024-01-27T04:20:25.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 15 client.18923.objecter _session_command_op_remove -1 2 2024-01-27T04:20:25.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 20 client.18923.objecter get_session s=0x7f44600740a0 osd=5 3 2024-01-27T04:20:25.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 15 client.18923.objecter _session_command_op_assign 5 2 2024-01-27T04:20:25.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 20 client.18923.objecter put_session s=0x7f44600740a0 osd=5 4 2024-01-27T04:20:25.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 10 client.18923.objecter _send_command 2 2024-01-27T04:20:25.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.262+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 --> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f44740aff40 con 0x7f4460074770 2024-01-27T04:20:25.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f44717fa640 1 -- 172.21.15.110:0/3195637301 <== osd.5 v2:172.21.15.162:6802/946981202 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f44740aff40 con 0x7f4460074770 2024-01-27T04:20:25.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f44717fa640 10 client.18923.objecter ms_dispatch 0x7f4474000b90 command_reply(tid 2: 0 ) v1 2024-01-27T04:20:25.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f44717fa640 10 client.18923.objecter _finish_command 2 = system:0 2024-01-27T04:20:25.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f44717fa640 20 client.18923.objecter put_session s=0x7f44600740a0 osd=5 4 2024-01-27T04:20:25.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f44717fa640 15 client.18923.objecter _session_command_op_remove 5 2 2024-01-27T04:20:25.277 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-01-27T04:20:25.277 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-27T04:20:25.277 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-01-27T04:20:25.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 10 librados: watch_flush enter 2024-01-27T04:20:25.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 10 librados: watch_flush exit 2024-01-27T04:20:25.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 10 client.18923.objecter close_session for osd.5 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f4460074770 msgr2=0x7f4460076c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6802/946981202,v1:172.21.15.162:6803/946981202] conn(0x7f4460074770 0x7f4460076c10 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 20 client.18923.objecter put_session s=0x7f44600740a0 osd=5 1 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 20 client.18923.objecter shutdown clearing up homeless session... 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 10 client.18923.objecter successfully canceled tick 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f446006c9f0 msgr2=0x7f446006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f446006c9f0 0x7f446006eeb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f44740b4560 tx=0x7f4468009190 comp rx=0 tx=0).stop 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 10 monclient: shutdown 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f447406a940 msgr2=0x7f44740adca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 --2- 172.21.15.110:0/3195637301 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f447406a940 0x7f44740adca0 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7f446400c440 tx=0x7f446400c900 comp rx=0 tx=0).stop 2024-01-27T04:20:25.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447931a640 1 -- 172.21.15.110:0/3195637301 reap_dead start 2024-01-27T04:20:25.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 shutdown_connections 2024-01-27T04:20:25.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 >> 172.21.15.110:0/3195637301 conn(0x7f44740b7d70 msgr2=0x7f44740b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:25.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 shutdown_connections 2024-01-27T04:20:25.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 -- 172.21.15.110:0/3195637301 wait complete. 2024-01-27T04:20:25.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.275+0000 7f447ad7d640 1 librados: shutdown 2024-01-27T04:20:25.287 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-27T04:20:25.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.386+0000 7fb3cddf5640 10 monclient: get_monmap_and_config 2024-01-27T04:20:25.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.386+0000 7fb3cddf5640 10 monclient: build_initial_monmap 2024-01-27T04:20:25.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.386+0000 7fb3cddf5640 10 monclient: monmap: 2024-01-27T04:20:25.388 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:25.388 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.388 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:25.387786+0000 2024-01-27T04:20:25.388 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:25.387786+0000 2024-01-27T04:20:25.388 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:25.388 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.388 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:25.389 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:25.389 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:25.389 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:25.389 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:25.389 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.387+0000 7fb3cddf5640 1 Processor -- start 2024-01-27T04:20:25.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.387+0000 7fb3cddf5640 1 -- start start 2024-01-27T04:20:25.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.387+0000 7fb3cddf5640 10 monclient: init 2024-01-27T04:20:25.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.387+0000 7fb3cddf5640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:25.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.387+0000 7fb3cddf5640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.387+0000 7fb3cddf5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3c80af7c0 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.387+0000 7fb3cddf5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb3c80abd10 0x7fb3c80aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 10 monclient(hunting): picked mon.noname-a con 0x7fb3c80abd10 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80ad3d0 0x7fb3c813c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 10 monclient(hunting): picked mon.noname-b con 0x7fb3c80ad3d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c80ad860 con 0x7fb3c80af7c0 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c80ab520 con 0x7fb3c80abd10 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8072480 con 0x7fb3c80ad3d0 2024-01-27T04:20:25.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3cddf5640 10 monclient(hunting): authenticate will time out at 3708.741999s 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3c6ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb3c80abd10 0x7fb3c80aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3c7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80ad3d0 0x7fb3c813c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3c7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80ad3d0 0x7fb3c813c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40828/0 (socket says 172.21.15.110:40828) 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3c7fff640 1 -- 172.21.15.110:0/3262292142 learned_addr learned my addr 172.21.15.110:0/3262292142 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3c7fff640 10 monclient(hunting): get_auth_request con 0x7fb3c80ad3d0 auth_method 0 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.388+0000 7fb3c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.389+0000 7fb3c6ffd640 10 monclient(hunting): get_auth_request con 0x7fb3c80abd10 auth_method 0 2024-01-27T04:20:25.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.389+0000 7fb3c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.389+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.389+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.389+0000 7fb3c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.389+0000 7fb3c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.389+0000 7fb3c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.390+0000 7fb3c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.390+0000 7fb3c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.390+0000 7fb3c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.391+0000 7fb3c7fff640 10 monclient(hunting): handle_auth_done global_id 18926 payload 274 2024-01-27T04:20:25.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.391+0000 7fb3c7fff640 1 -- 172.21.15.110:0/3262292142 >> v2:172.21.15.110:3300/0 conn(0x7fb3c80abd10 msgr2=0x7fb3c80aaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:25.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.391+0000 7fb3c7fff640 1 --2- 172.21.15.110:0/3262292142 >> v2:172.21.15.110:3300/0 conn(0x7fb3c80abd10 0x7fb3c80aaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.391+0000 7fb3c7fff640 1 -- 172.21.15.110:0/3262292142 >> v1:172.21.15.110:6789/0 conn(0x7fb3c80af7c0 legacy=0x7fb3c8071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.391+0000 7fb3c7fff640 10 monclient: _finish_hunting 0 2024-01-27T04:20:25.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.391+0000 7fb3c7fff640 1 monclient: found mon.noname-b 2024-01-27T04:20:25.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.391+0000 7fb3c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:25.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.391+0000 7fb3c7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.391+0000 7fb3c7fff640 1 -- 172.21.15.110:0/3262292142 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c80aab90 con 0x7fb3c80ad3d0 2024-01-27T04:20:25.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.393+0000 7fb3c7fff640 1 --2- 172.21.15.110:0/3262292142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80ad3d0 0x7fb3c813c1c0 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000d150 tx=0x7fb3c002e0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c9c493ebead65f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 1 -- 172.21.15.110:0/3262292142 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c002ea80 con 0x7fb3c80ad3d0 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 10 monclient: dump: 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:25.398 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 10 monclient: _finish_auth 0 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 10 monclient: _check_auth_tickets 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 1 -- 172.21.15.110:0/3262292142 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c002ebe0 con 0x7fb3c80ad3d0 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3cddf5640 5 monclient: authenticate success, global_id 18926 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3cddf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3cddf5640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:25.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3cddf5640 4 set_mon_vals no callback set 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 1 -- 172.21.15.110:0/3262292142 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0009600 con 0x7fb3c80ad3d0 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.396+0000 7fb3c67fc640 10 monclient: dump: 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:25.400 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:25.401 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 10 monclient: shutdown 2024-01-27T04:20:25.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:25.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 1 -- 172.21.15.110:0/3262292142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80ad3d0 msgr2=0x7fb3c813c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 1 --2- 172.21.15.110:0/3262292142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80ad3d0 0x7fb3c813c1c0 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000d150 tx=0x7fb3c002e0e0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 1 -- 172.21.15.110:0/3262292142 shutdown_connections 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 1 --2- 172.21.15.110:0/3262292142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80ad3d0 0x7fb3c813c1c0 unknown :-1 s=CLOSED pgs=3419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 1 --2- 172.21.15.110:0/3262292142 >> v2:172.21.15.110:3300/0 conn(0x7fb3c80abd10 0x7fb3c80aaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 1 -- 172.21.15.110:0/3262292142 >> 172.21.15.110:0/3262292142 conn(0x7fb3c80b3a40 msgr2=0x7fb3c80b3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 1 -- 172.21.15.110:0/3262292142 shutdown_connections 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.397+0000 7fb3cddf5640 1 -- 172.21.15.110:0/3262292142 wait complete. 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.398+0000 7fb3cddf5640 10 monclient: build_initial_monmap 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.398+0000 7fb3cddf5640 10 monclient: monmap: 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:25.405 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.398+0000 7fb3cddf5640 1 librados: starting msgr at 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.398+0000 7fb3cddf5640 1 librados: starting objecter 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.398+0000 7fb3cddf5640 1 Processor -- start 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.398+0000 7fb3cddf5640 1 -- start start 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.398+0000 7fb3cddf5640 1 librados: setting wanted keys 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.398+0000 7fb3cddf5640 1 librados: calling monclient init 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.398+0000 7fb3cddf5640 10 monclient: init 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T04:20:25.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3c80abd10 0x7fb3c806df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient(hunting): picked mon.noname-b con 0x7fb3c80abd10 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c80ad3d0 0x7fb3c8070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient(hunting): picked mon.noname-a con 0x7fb3c80ad3d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80af7c0 0x7fb3c8070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient(hunting): picked mon.noname-c con 0x7fb3c80af7c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c806fd60 con 0x7fb3c80ad3d0 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8070040 con 0x7fb3c80abd10 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8070320 con 0x7fb3c80af7c0 2024-01-27T04:20:25.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3cddf5640 10 monclient(hunting): authenticate will time out at 3708.753468s 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3c7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80af7c0 0x7fb3c8070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.399+0000 7fb3c6ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c80ad3d0 0x7fb3c8070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80af7c0 0x7fb3c8070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40830/0 (socket says 172.21.15.110:40830) 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c7fff640 1 -- 172.21.15.110:0/2159063979 learned_addr learned my addr 172.21.15.110:0/2159063979 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c7fff640 10 monclient(hunting): get_auth_request con 0x7fb3c80af7c0 auth_method 0 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c77fe640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3c80abd10 0x7fb3c806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c6ffd640 10 monclient(hunting): get_auth_request con 0x7fb3c80ad3d0 auth_method 0 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c77fe640 10 monclient(hunting): get_auth_request con 0x7fb3c80abd10 auth_method 0 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.400+0000 7fb3c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.401+0000 7fb3c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.401+0000 7fb3c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.401+0000 7fb3c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:25.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 10 monclient(hunting): handle_auth_done global_id 18929 payload 824 2024-01-27T04:20:25.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 1 -- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3c80abd10 msgr2=0x7fb3c806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3c80abd10 0x7fb3c806df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 1 -- 172.21.15.110:0/2159063979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c80ad3d0 msgr2=0x7fb3c8070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c80ad3d0 0x7fb3c8070a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 10 monclient: _finish_hunting 0 2024-01-27T04:20:25.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 1 monclient: found mon.noname-c 2024-01-27T04:20:25.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:25.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.403+0000 7fb3c7fff640 1 -- 172.21.15.110:0/2159063979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c000bd40 con 0x7fb3c80af7c0 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.406+0000 7fb3c7fff640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80af7c0 0x7fb3c8070fc0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000d150 tx=0x7fb3c0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.406+0000 7fb3affff640 10 client.?.objecter ms_handle_connect 0x7fb3c80af7c0 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.406+0000 7fb3affff640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 1 -- 172.21.15.110:0/2159063979 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c002ea80 con 0x7fb3c80af7c0 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 10 monclient: dump: 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.412 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 10 monclient: _finish_auth 0 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 10 monclient: _check_auth_tickets 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 1 -- 172.21.15.110:0/2159063979 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c000b720 con 0x7fb3c80af7c0 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3cddf5640 5 monclient: authenticate success, global_id 18929 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3cddf5640 10 monclient: _renew_subs 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3cddf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c806f0e0 con 0x7fb3c80af7c0 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:25.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 1 -- 172.21.15.110:0/2159063979 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c00408a0 con 0x7fb3c80af7c0 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3affff640 10 monclient: dump: 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3c5ffb640 4 set_mon_vals no callback set 2024-01-27T04:20:25.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3cddf5640 10 client.18929.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3cddf5640 10 monclient: _renew_subs 2024-01-27T04:20:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3cddf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c806eb60 con 0x7fb3c80af7c0 2024-01-27T04:20:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.409+0000 7fb3cddf5640 1 librados: init done 2024-01-27T04:20:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.410+0000 7fb3cddf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.410+0000 7fb3cddf5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.410+0000 7fb3cddf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:25.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.410+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c8070740 con 0x7fb3c80af7c0 2024-01-27T04:20:25.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.658+0000 7fb3affff640 1 -- 172.21.15.110:0/2159063979 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fb3c002ed50 con 0x7fb3c80af7c0 2024-01-27T04:20:25.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.659+0000 7fb3affff640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:25.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.659+0000 7fb3c77fe640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:25.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.659+0000 7fb3c77fe640 10 monclient: get_auth_request con 0x7fb3b406c9f0 auth_method 0 2024-01-27T04:20:25.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.660+0000 7fb3c77fe640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb3b80016e0 tx=0x7fb3b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:25.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.660+0000 7fb3affff640 10 client.18929.objecter ms_handle_connect 0x7fb3b406c9f0 2024-01-27T04:20:25.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.671+0000 7fb3affff640 1 -- 172.21.15.110:0/2159063979 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7fb3c0009e30 con 0x7fb3c80af7c0 2024-01-27T04:20:25.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.671+0000 7fb3affff640 10 client.18929.objecter ms_dispatch 0x7fb3c8000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:25.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.671+0000 7fb3affff640 3 client.18929.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:25.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.671+0000 7fb3affff640 3 client.18929.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:25.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.672+0000 7fb3affff640 20 client.18929.objecter dump_active .. 0 homeless 2024-01-27T04:20:25.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.672+0000 7fb3affff640 1 -- 172.21.15.110:0/2159063979 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fb3c0079840 con 0x7fb3c80af7c0 2024-01-27T04:20:25.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.672+0000 7fb3affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:25.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.672+0000 7fb3affff640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:25.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.880+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3c80ae440 con 0x7fb3b406c9f0 2024-01-27T04:20:25.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.880+0000 7fb3affff640 1 -- 172.21.15.110:0/2159063979 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb3b8001cc0 con 0x7fb3b406c9f0 2024-01-27T04:20:25.882 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 5 to deep-scrub 2024-01-27T04:20:25.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 10 librados: watch_flush enter 2024-01-27T04:20:25.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 10 librados: watch_flush exit 2024-01-27T04:20:25.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 20 client.18929.objecter shutdown clearing up homeless session... 2024-01-27T04:20:25.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 10 client.18929.objecter successfully canceled tick 2024-01-27T04:20:25.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 msgr2=0x7fb3b406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb3b80016e0 tx=0x7fb3b8009040 comp rx=0 tx=0).stop 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 10 monclient: shutdown 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80af7c0 msgr2=0x7fb3c8070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80af7c0 0x7fb3c8070fc0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000d150 tx=0x7fb3c0004a40 comp rx=0 tx=0).stop 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 shutdown_connections 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fb3b406c9f0 0x7fb3b406eeb0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c80af7c0 0x7fb3c8070fc0 unknown :-1 s=CLOSED pgs=3420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c80ad3d0 0x7fb3c8070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 --2- 172.21.15.110:0/2159063979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3c80abd10 0x7fb3c806df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.882+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 >> 172.21.15.110:0/2159063979 conn(0x7fb3c80b3a40 msgr2=0x7fb3c80aeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.883+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 shutdown_connections 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.883+0000 7fb3cddf5640 1 -- 172.21.15.110:0/2159063979 wait complete. 2024-01-27T04:20:25.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:25.883+0000 7fb3cddf5640 1 librados: shutdown 2024-01-27T04:20:25.896 INFO:tasks.ceph:Scrubbing osd.6 2024-01-27T04:20:25.897 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-27T04:20:26.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.003+0000 7f63b71e2640 10 monclient: get_monmap_and_config 2024-01-27T04:20:26.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.003+0000 7f63b71e2640 10 monclient: build_initial_monmap 2024-01-27T04:20:26.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.003+0000 7f63b71e2640 10 monclient: monmap: 2024-01-27T04:20:26.004 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:26.004261+0000 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:26.004261+0000 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:26.005 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:26.006 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.003+0000 7f63b71e2640 1 Processor -- start 2024-01-27T04:20:26.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.003+0000 7f63b71e2640 1 -- start start 2024-01-27T04:20:26.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.003+0000 7f63b71e2640 10 monclient: init 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f63b006a940 0x7f63b01426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 10 monclient(hunting): picked mon.noname-a con 0x7f63b006a940 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b00a9e40 0x7f63b0142c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 10 monclient(hunting): picked mon.noname-b con 0x7f63b00a9e40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63b00b2d40 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b00aac50 con 0x7f63b00b2d40 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b00b38a0 con 0x7f63b006a940 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b006ad20 con 0x7f63b00a9e40 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:26.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.004+0000 7f63b71e2640 10 monclient(hunting): authenticate will time out at 3709.358458s 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63affff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b00a9e40 0x7f63b0142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63b4f7e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f63b006a940 0x7f63b01426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63b577f640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f63b00b2d40 0x7f63b00b3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.110:34642/0 (socket says 172.21.15.110:34642) 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63b577f640 1 -- 172.21.15.110:0/1012968072 learned_addr learned my addr 172.21.15.110:0/1012968072 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63affff640 10 monclient(hunting): get_auth_request con 0x7f63b00a9e40 auth_method 0 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63affff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63affff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63b4f7e640 10 monclient(hunting): get_auth_request con 0x7f63b006a940 auth_method 0 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63b4f7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63b4f7e640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.005+0000 7f63b4f7e640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.006+0000 7f63affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.006+0000 7f63affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.006+0000 7f63affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.006+0000 7f63b4f7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.006+0000 7f63b4f7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.006+0000 7f63b4f7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.007+0000 7f63af7fe640 1 -- 172.21.15.110:0/1012968072 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019909806 0 0) 0x7f63b00aac50 con 0x7f63b00b2d40 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.007+0000 7f63af7fe640 10 monclient(hunting): _negotiate 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.007+0000 7f63af7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.007+0000 7f63af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.007+0000 7f63af7fe640 10 monclient(hunting): my global_id is 20974 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.007+0000 7f63af7fe640 1 -- 172.21.15.110:0/1012968072 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a0003520 con 0x7f63b00b2d40 2024-01-27T04:20:26.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.007+0000 7f63affff640 10 monclient(hunting): handle_auth_done global_id 18932 payload 274 2024-01-27T04:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.008+0000 7f63affff640 1 -- 172.21.15.110:0/1012968072 >> v2:172.21.15.110:3300/0 conn(0x7f63b006a940 msgr2=0x7f63b01426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.008+0000 7f63affff640 1 --2- 172.21.15.110:0/1012968072 >> v2:172.21.15.110:3300/0 conn(0x7f63b006a940 0x7f63b01426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.008+0000 7f63affff640 1 -- 172.21.15.110:0/1012968072 >> v1:172.21.15.162:6789/0 conn(0x7f63b00b2d40 legacy=0x7f63b00b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.008+0000 7f63affff640 10 monclient: _finish_hunting 0 2024-01-27T04:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.008+0000 7f63affff640 1 monclient: found mon.noname-b 2024-01-27T04:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.008+0000 7f63affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.008+0000 7f63affff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:26.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.008+0000 7f63affff640 1 -- 172.21.15.110:0/1012968072 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b00aaf30 con 0x7f63b00a9e40 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.010+0000 7f63affff640 1 --2- 172.21.15.110:0/1012968072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b00a9e40 0x7f63b0142c10 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f639c0097a0 tx=0x7f639c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcb91320c24d3fe8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 1 -- 172.21.15.110:0/1012968072 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c031830 con 0x7f63b00a9e40 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 10 monclient: dump: 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.015 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 10 monclient: _finish_auth 0 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 10 monclient: _check_auth_tickets 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 1 -- 172.21.15.110:0/1012968072 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c031990 con 0x7f63b00a9e40 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 1 -- 172.21.15.110:0/1012968072 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c031d20 con 0x7f63b00a9e40 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63b71e2640 5 monclient: authenticate success, global_id 18932 2024-01-27T04:20:26.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63b71e2640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63b71e2640 4 set_mon_vals no callback set 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.013+0000 7f63af7fe640 10 monclient: dump: 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.017 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.018 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 10 monclient: shutdown 2024-01-27T04:20:26.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:26.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 1 -- 172.21.15.110:0/1012968072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b00a9e40 msgr2=0x7f63b0142c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 1 --2- 172.21.15.110:0/1012968072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b00a9e40 0x7f63b0142c10 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f639c0097a0 tx=0x7f639c02b8d0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 1 -- 172.21.15.110:0/1012968072 shutdown_connections 2024-01-27T04:20:26.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 1 --2- 172.21.15.110:0/1012968072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b00a9e40 0x7f63b0142c10 unknown :-1 s=CLOSED pgs=3421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 1 --2- 172.21.15.110:0/1012968072 >> v2:172.21.15.110:3300/0 conn(0x7f63b006a940 0x7f63b01426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 1 -- 172.21.15.110:0/1012968072 >> 172.21.15.110:0/1012968072 conn(0x7f63b00b7d70 msgr2=0x7f63b00b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 1 -- 172.21.15.110:0/1012968072 shutdown_connections 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.014+0000 7f63b71e2640 1 -- 172.21.15.110:0/1012968072 wait complete. 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.015+0000 7f63b71e2640 10 monclient: build_initial_monmap 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.015+0000 7f63b71e2640 10 monclient: monmap: 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.015+0000 7f63b71e2640 1 librados: starting msgr at 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.015+0000 7f63b71e2640 1 librados: starting objecter 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 Processor -- start 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 -- start start 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 librados: setting wanted keys 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 librados: calling monclient init 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient: init 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b006a940 0x7f63b01495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient(hunting): picked mon.noname-c con 0x7f63b006a940 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63b00a9e40 0x7f63b014bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient(hunting): picked mon.noname-a con 0x7f63b00a9e40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b00b2d40 0x7f63b014c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient(hunting): picked mon.noname-b con 0x7f63b00b2d40 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b014ada0 con 0x7f63b00a9e40 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b014c870 con 0x7f63b00b2d40 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b014afa0 con 0x7f63b006a940 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.016+0000 7f63b71e2640 10 monclient(hunting): authenticate will time out at 3709.370800s 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b4f7e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b006a940 0x7f63b01495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63affff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63b00a9e40 0x7f63b014bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b577f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b00b2d40 0x7f63b014c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b4f7e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b006a940 0x7f63b01495e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40840/0 (socket says 172.21.15.110:40840) 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b4f7e640 1 -- 172.21.15.110:0/816467358 learned_addr learned my addr 172.21.15.110:0/816467358 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:26.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b4f7e640 10 monclient(hunting): get_auth_request con 0x7f63b006a940 auth_method 0 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b4f7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b4f7e640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b4f7e640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63affff640 10 monclient(hunting): get_auth_request con 0x7f63b00a9e40 auth_method 0 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63affff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63affff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b577f640 10 monclient(hunting): get_auth_request con 0x7f63b00b2d40 auth_method 0 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b577f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b577f640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.017+0000 7f63b577f640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.018+0000 7f63b4f7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.018+0000 7f63b4f7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.018+0000 7f63b4f7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.018+0000 7f63affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.018+0000 7f63affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.018+0000 7f63affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.018+0000 7f63b577f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.018+0000 7f63b577f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.018+0000 7f63b577f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.020+0000 7f63b577f640 10 monclient(hunting): handle_auth_done global_id 20977 payload 824 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.021+0000 7f63b577f640 1 -- 172.21.15.110:0/816467358 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b006a940 msgr2=0x7f63b01495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.021+0000 7f63b577f640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63b006a940 0x7f63b01495e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.021+0000 7f63b577f640 1 -- 172.21.15.110:0/816467358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63b00a9e40 msgr2=0x7f63b014bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.021+0000 7f63b577f640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63b00a9e40 0x7f63b014bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.021+0000 7f63b577f640 10 monclient: _finish_hunting 0 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.021+0000 7f63b577f640 1 monclient: found mon.noname-b 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.021+0000 7f63b577f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.021+0000 7f63b577f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:26.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.021+0000 7f63b577f640 1 -- 172.21.15.110:0/816467358 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c009400 con 0x7f63b00b2d40 2024-01-27T04:20:26.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.023+0000 7f63b577f640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b00b2d40 0x7f63b014c330 secure :-1 s=READY pgs=5100 cs=0 l=1 rev1=1 crypto rx=0x7f63a800caa0 tx=0x7f63a800cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.023+0000 7f63ad7fa640 10 client.?.objecter ms_handle_connect 0x7f63b00b2d40 2024-01-27T04:20:26.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.023+0000 7f63ad7fa640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:26.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 1 -- 172.21.15.110:0/816467358 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a800ab40 con 0x7f63b00b2d40 2024-01-27T04:20:26.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:26.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 10 monclient: dump: 2024-01-27T04:20:26.029 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.029 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 10 monclient: _finish_auth 0 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 10 monclient: _check_auth_tickets 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 1 -- 172.21.15.110:0/816467358 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a800aca0 con 0x7f63b00b2d40 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 1 -- 172.21.15.110:0/816467358 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8012600 con 0x7f63b00b2d40 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63b71e2640 5 monclient: authenticate success, global_id 20977 2024-01-27T04:20:26.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63b71e2640 10 monclient: _renew_subs 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63b71e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b01467c0 con 0x7f63b00b2d40 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63ad7fa640 10 monclient: dump: 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.031 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63aeffd640 4 set_mon_vals no callback set 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63b71e2640 10 client.20977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63b71e2640 10 monclient: _renew_subs 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63b71e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b0149d00 con 0x7f63b00b2d40 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.026+0000 7f63b71e2640 1 librados: init done 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.027+0000 7f63b71e2640 10 client.20977.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.027+0000 7f63b71e2640 15 client.20977.objecter _session_command_op_assign -1 1 2024-01-27T04:20:26.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.027+0000 7f63b71e2640 20 client.20977.objecter _get_session osd=-1 returning homeless 2024-01-27T04:20:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.027+0000 7f63b71e2640 10 client.20977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.027+0000 7f63b71e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:26.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.027+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63b00b4590 con 0x7f63b00b2d40 2024-01-27T04:20:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.045+0000 7f63ad7fa640 1 -- 172.21.15.110:0/816467358 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f63a8007070 con 0x7f63b00b2d40 2024-01-27T04:20:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.045+0000 7f63ad7fa640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639406c9f0 0x7f639406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.045+0000 7f63b4f7e640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639406c9f0 0x7f639406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.045+0000 7f63b4f7e640 10 monclient: get_auth_request con 0x7f639406c9f0 auth_method 0 2024-01-27T04:20:26.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.046+0000 7f63b4f7e640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639406c9f0 0x7f639406eeb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009c00 tx=0x7f63a4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.046+0000 7f63ad7fa640 10 client.20977.objecter ms_handle_connect 0x7f639406c9f0 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 1 -- 172.21.15.110:0/816467358 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f63a8058a80 con 0x7f63b00b2d40 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 10 client.20977.objecter ms_dispatch 0x7f63b0000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 3 client.20977.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 3 client.20977.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 10 client.20977.objecter checking command 1 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f6394074770 0x7f6394076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 20 client.20977.objecter _get_session s=0x7f63940740a0 osd=6 3 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 20 client.20977.objecter put_session s=0x7f63940740a0 osd=6 3 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 15 client.20977.objecter _session_command_op_remove -1 1 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 20 client.20977.objecter _get_session s=0x7f63940740a0 osd=6 3 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 20 client.20977.objecter get_session s=0x7f63940740a0 osd=6 3 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 15 client.20977.objecter _session_command_op_assign 6 1 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 20 client.20977.objecter put_session s=0x7f63940740a0 osd=6 4 2024-01-27T04:20:26.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 10 client.20977.objecter _send_command 1 2024-01-27T04:20:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 1 -- 172.21.15.110:0/816467358 --> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63940772e0 con 0x7f6394074770 2024-01-27T04:20:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63ad7fa640 20 client.20977.objecter dump_active .. 0 homeless 2024-01-27T04:20:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63affff640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f6394074770 0x7f6394076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.049+0000 7f63affff640 10 monclient: get_auth_request con 0x7f6394074770 auth_method 0 2024-01-27T04:20:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.050+0000 7f63ad7fa640 1 -- 172.21.15.110:0/816467358 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=1334) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63a8054d20 con 0x7f63b00b2d40 2024-01-27T04:20:26.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.050+0000 7f63ad7fa640 10 monclient: handle_get_version_reply finishing 1 version 1334 2024-01-27T04:20:26.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.052+0000 7f63affff640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f6394074770 0x7f6394076c10 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.052+0000 7f63ad7fa640 10 client.20977.objecter ms_handle_connect 0x7f6394074770 2024-01-27T04:20:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.076+0000 7f63ad7fa640 1 -- 172.21.15.110:0/816467358 <== osd.6 v2:172.21.15.162:6810/3628034141 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f63940772e0 con 0x7f6394074770 2024-01-27T04:20:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.076+0000 7f63ad7fa640 10 client.20977.objecter ms_dispatch 0x7f63b0000b90 command_reply(tid 1: 0 ) v1 2024-01-27T04:20:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.076+0000 7f63ad7fa640 10 client.20977.objecter _finish_command 1 = system:0 2024-01-27T04:20:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.076+0000 7f63ad7fa640 20 client.20977.objecter put_session s=0x7f63940740a0 osd=6 4 2024-01-27T04:20:26.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.076+0000 7f63ad7fa640 15 client.20977.objecter _session_command_op_remove 6 1 2024-01-27T04:20:26.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 10 client.20977.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-27T04:20:26.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 15 client.20977.objecter _session_command_op_assign -1 2 2024-01-27T04:20:26.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 20 client.20977.objecter _get_session s=0x7f63940740a0 osd=6 3 2024-01-27T04:20:26.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 20 client.20977.objecter put_session s=0x7f63940740a0 osd=6 3 2024-01-27T04:20:26.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 20 client.20977.objecter _get_session s=0x7f63940740a0 osd=6 3 2024-01-27T04:20:26.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 15 client.20977.objecter _session_command_op_remove -1 2 2024-01-27T04:20:26.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 20 client.20977.objecter get_session s=0x7f63940740a0 osd=6 3 2024-01-27T04:20:26.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 15 client.20977.objecter _session_command_op_assign 6 2 2024-01-27T04:20:26.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 20 client.20977.objecter put_session s=0x7f63940740a0 osd=6 4 2024-01-27T04:20:26.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 10 client.20977.objecter _send_command 2 2024-01-27T04:20:26.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.090+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 --> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f63b0074d10 con 0x7f6394074770 2024-01-27T04:20:26.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.105+0000 7f63ad7fa640 1 -- 172.21.15.110:0/816467358 <== osd.6 v2:172.21.15.162:6810/3628034141 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f63b0074d10 con 0x7f6394074770 2024-01-27T04:20:26.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.105+0000 7f63ad7fa640 10 client.20977.objecter ms_dispatch 0x7f63b0000b90 command_reply(tid 2: 0 ) v1 2024-01-27T04:20:26.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.105+0000 7f63ad7fa640 10 client.20977.objecter _finish_command 2 = system:0 2024-01-27T04:20:26.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.105+0000 7f63ad7fa640 20 client.20977.objecter put_session s=0x7f63940740a0 osd=6 4 2024-01-27T04:20:26.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.105+0000 7f63ad7fa640 15 client.20977.objecter _session_command_op_remove 6 2 2024-01-27T04:20:26.108 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-01-27T04:20:26.108 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-27T04:20:26.108 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-01-27T04:20:26.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 10 librados: watch_flush enter 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 10 librados: watch_flush exit 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 10 client.20977.objecter close_session for osd.6 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f6394074770 msgr2=0x7f6394076c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6810/3628034141,v1:172.21.15.162:6811/3628034141] conn(0x7f6394074770 0x7f6394076c10 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 20 client.20977.objecter put_session s=0x7f63940740a0 osd=6 1 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 20 client.20977.objecter shutdown clearing up homeless session... 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 10 client.20977.objecter successfully canceled tick 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639406c9f0 msgr2=0x7f639406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f639406c9f0 0x7f639406eeb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009c00 tx=0x7f63a4009190 comp rx=0 tx=0).stop 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 10 monclient: shutdown 2024-01-27T04:20:26.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:26.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b00b2d40 msgr2=0x7f63b014c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 --2- 172.21.15.110:0/816467358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b00b2d40 0x7f63b014c330 secure :-1 s=READY pgs=5100 cs=0 l=1 rev1=1 crypto rx=0x7f63a800caa0 tx=0x7f63a800cea0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b577f640 1 -- 172.21.15.110:0/816467358 reap_dead start 2024-01-27T04:20:26.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 shutdown_connections 2024-01-27T04:20:26.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 >> 172.21.15.110:0/816467358 conn(0x7f63b00b7d70 msgr2=0x7f63b01414e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:26.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 shutdown_connections 2024-01-27T04:20:26.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 -- 172.21.15.110:0/816467358 wait complete. 2024-01-27T04:20:26.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.106+0000 7f63b71e2640 1 librados: shutdown 2024-01-27T04:20:26.118 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-27T04:20:26.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.216+0000 7fab02487640 10 monclient: get_monmap_and_config 2024-01-27T04:20:26.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.216+0000 7fab02487640 10 monclient: build_initial_monmap 2024-01-27T04:20:26.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.216+0000 7fab02487640 10 monclient: monmap: 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:26.217200+0000 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:26.217200+0000 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:26.218 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:26.219 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.216+0000 7fab02487640 1 Processor -- start 2024-01-27T04:20:26.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.216+0000 7fab02487640 1 -- start start 2024-01-27T04:20:26.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.216+0000 7fab02487640 10 monclient: init 2024-01-27T04:20:26.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:26.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-01-27T04:20:26.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7faafc0b4680 0x7faafc142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient(hunting): picked mon.noname-c con 0x7faafc0b4680 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:26.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faafc0b4a60 0x7faafc0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient(hunting): picked mon.noname-a con 0x7faafc0b4a60 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc142eb0 0x7faafc0b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient(hunting): picked mon.noname-b con 0x7faafc142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faafc142240 con 0x7faafc0b4a60 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faafc0afbe0 con 0x7faafc0b4680 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faafc0b0890 con 0x7faafc142eb0 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.217+0000 7fab02487640 10 monclient(hunting): authenticate will time out at 3709.571412s 2024-01-27T04:20:26.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faafc0b4a60 0x7faafc0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafbfff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7faafc0b4680 0x7faafc142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faafc0b4a60 0x7faafc0b1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45046/0 (socket says 172.21.15.110:45046) 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 1 -- 172.21.15.110:0/4137975582 learned_addr learned my addr 172.21.15.110:0/4137975582 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 10 monclient(hunting): get_auth_request con 0x7faafc0b4a60 auth_method 0 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafbfff640 10 monclient(hunting): get_auth_request con 0x7faafc0b4680 auth_method 0 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafbfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.218+0000 7faafb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.219+0000 7faafbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.219+0000 7faafbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.219+0000 7faafbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.219+0000 7fab00a24640 1 --2- 172.21.15.110:0/4137975582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc142eb0 0x7faafc0b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.219+0000 7fab00a24640 10 monclient(hunting): get_auth_request con 0x7faafc142eb0 auth_method 0 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.219+0000 7fab00a24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.219+0000 7fab00a24640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.219+0000 7fab00a24640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 10 monclient(hunting): handle_auth_done global_id 17073 payload 274 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 1 -- 172.21.15.110:0/4137975582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc142eb0 msgr2=0x7faafc0b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 1 --2- 172.21.15.110:0/4137975582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc142eb0 0x7faafc0b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 1 -- 172.21.15.110:0/4137975582 >> v2:172.21.15.162:3300/0 conn(0x7faafc0b4680 msgr2=0x7faafc142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 1 --2- 172.21.15.110:0/4137975582 >> v2:172.21.15.162:3300/0 conn(0x7faafc0b4680 0x7faafc142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 10 monclient: _finish_hunting 0 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 1 monclient: found mon.noname-a 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7faafb7fe640 1 -- 172.21.15.110:0/4137975582 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc0b0b00 con 0x7faafc0b4a60 2024-01-27T04:20:26.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.220+0000 7fab00a24640 1 --2- 172.21.15.110:0/4137975582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc142eb0 0x7faafc0b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-27T04:20:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.222+0000 7faafb7fe640 1 --2- 172.21.15.110:0/4137975582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc0b4a60 0x7faafc0b1470 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7faafc0b4ee0 tx=0x7faaf002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b20dd2328ab816e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 1 -- 172.21.15.110:0/4137975582 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf00317d0 con 0x7faafc0b4a60 2024-01-27T04:20:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T04:20:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 10 monclient: dump: 2024-01-27T04:20:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.228 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.229 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 10 monclient: _finish_auth 0 2024-01-27T04:20:26.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 10 monclient: _check_auth_tickets 2024-01-27T04:20:26.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:26.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 1 -- 172.21.15.110:0/4137975582 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf0031930 con 0x7faafc0b4a60 2024-01-27T04:20:26.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 5 monclient: authenticate success, global_id 17073 2024-01-27T04:20:26.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:26.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:26.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 4 set_mon_vals no callback set 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 1 -- 172.21.15.110:0/4137975582 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0031cc0 con 0x7faafc0b4a60 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7faafaffd640 10 monclient: dump: 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.230 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.231 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.231 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.231 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.231 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.231 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 10 monclient: shutdown 2024-01-27T04:20:26.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:26.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 1 -- 172.21.15.110:0/4137975582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc0b4a60 msgr2=0x7faafc0b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 1 --2- 172.21.15.110:0/4137975582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc0b4a60 0x7faafc0b1470 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7faafc0b4ee0 tx=0x7faaf002b870 comp rx=0 tx=0).stop 2024-01-27T04:20:26.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 1 -- 172.21.15.110:0/4137975582 shutdown_connections 2024-01-27T04:20:26.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 1 --2- 172.21.15.110:0/4137975582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc142eb0 0x7faafc0b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 1 --2- 172.21.15.110:0/4137975582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc0b4a60 0x7faafc0b1470 unknown :-1 s=CLOSED pgs=3472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 1 --2- 172.21.15.110:0/4137975582 >> v2:172.21.15.162:3300/0 conn(0x7faafc0b4680 0x7faafc142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.226+0000 7fab02487640 1 -- 172.21.15.110:0/4137975582 >> 172.21.15.110:0/4137975582 conn(0x7faafc066490 msgr2=0x7faafc0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.227+0000 7fab02487640 1 -- 172.21.15.110:0/4137975582 shutdown_connections 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.227+0000 7fab02487640 1 -- 172.21.15.110:0/4137975582 wait complete. 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.227+0000 7fab02487640 10 monclient: build_initial_monmap 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.227+0000 7fab02487640 10 monclient: monmap: 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:26.236 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.227+0000 7fab02487640 1 librados: starting msgr at 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.227+0000 7fab02487640 1 librados: starting objecter 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 Processor -- start 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 -- start start 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 librados: setting wanted keys 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 librados: calling monclient init 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient: init 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc0b4680 0x7faafc0aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient(hunting): picked mon.noname-c con 0x7faafc0b4680 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc142eb0 0x7faafc1d6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient(hunting): picked mon.noname-a con 0x7faafc142eb0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:26.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faafc1d7480 0x7faafc0ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient(hunting): picked mon.noname-b con 0x7faafc1d7480 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc0ab8f0 con 0x7faafc142eb0 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc0aad50 con 0x7faafc1d7480 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faafc0abb40 con 0x7faafc0b4680 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.228+0000 7fab02487640 10 monclient(hunting): authenticate will time out at 3709.582820s 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc142eb0 0x7faafc1d6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc0b4680 0x7faafc0aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7fab00a24640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faafc1d7480 0x7faafc0ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc142eb0 0x7faafc1d6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45054/0 (socket says 172.21.15.110:45054) 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafb7fe640 1 -- 172.21.15.110:0/3399580179 learned_addr learned my addr 172.21.15.110:0/3399580179 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafb7fe640 10 monclient(hunting): get_auth_request con 0x7faafc142eb0 auth_method 0 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7fab00a24640 10 monclient(hunting): get_auth_request con 0x7faafc1d7480 auth_method 0 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7fab00a24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7fab00a24640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7fab00a24640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafbfff640 10 monclient(hunting): get_auth_request con 0x7faafc0b4680 auth_method 0 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafbfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.229+0000 7faafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.230+0000 7fab00a24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.230+0000 7fab00a24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.230+0000 7fab00a24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.230+0000 7faafb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.230+0000 7faafb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.230+0000 7faafb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.230+0000 7faafbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.230+0000 7faafbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.230+0000 7faafbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 10 monclient(hunting): handle_auth_done global_id 20983 payload 824 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 1 -- 172.21.15.110:0/3399580179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc0b4680 msgr2=0x7faafc0aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc0b4680 0x7faafc0aa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 1 -- 172.21.15.110:0/3399580179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc142eb0 msgr2=0x7faafc1d6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc142eb0 0x7faafc1d6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 10 monclient: _finish_hunting 0 2024-01-27T04:20:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 1 monclient: found mon.noname-b 2024-01-27T04:20:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:26.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.232+0000 7fab00a24640 1 -- 172.21.15.110:0/3399580179 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf00094d0 con 0x7faafc1d7480 2024-01-27T04:20:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.234+0000 7fab00a24640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faafc1d7480 0x7faafc0ad8f0 secure :-1 s=READY pgs=5101 cs=0 l=1 rev1=1 crypto rx=0x7faaf4007ec0 tx=0x7faaf400d4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.234+0000 7faaf8ff9640 10 client.?.objecter ms_handle_connect 0x7faafc1d7480 2024-01-27T04:20:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.234+0000 7faaf8ff9640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 1 -- 172.21.15.110:0/3399580179 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf4015600 con 0x7faafc1d7480 2024-01-27T04:20:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:26.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 10 monclient: dump: 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 10 monclient: _finish_auth 0 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 10 monclient: _check_auth_tickets 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:26.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 1 -- 172.21.15.110:0/3399580179 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf400f040 con 0x7faafc1d7480 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7fab02487640 5 monclient: authenticate success, global_id 20983 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7fab02487640 10 monclient: _renew_subs 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7fab02487640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faafc0abe20 con 0x7faafc1d7480 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 1 -- 172.21.15.110:0/3399580179 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf40159c0 con 0x7faafc1d7480 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faaf8ff9640 10 monclient: dump: 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.244 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7fab02487640 10 client.20983.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7fab02487640 10 monclient: _renew_subs 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7fab02487640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faafc1d7930 con 0x7faafc1d7480 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7faafa7fc640 4 set_mon_vals no callback set 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.238+0000 7fab02487640 1 librados: init done 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.239+0000 7fab02487640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.239+0000 7fab02487640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:26.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.239+0000 7fab02487640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:26.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.239+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faafc074d10 con 0x7faafc1d7480 2024-01-27T04:20:26.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.257+0000 7faaf8ff9640 1 -- 172.21.15.110:0/3399580179 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7faaf401f440 con 0x7faafc1d7480 2024-01-27T04:20:26.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.258+0000 7faaf8ff9640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7faae806c9f0 0x7faae806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.258+0000 7faafbfff640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7faae806c9f0 0x7faae806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.258+0000 7faafbfff640 10 monclient: get_auth_request con 0x7faae806c9f0 auth_method 0 2024-01-27T04:20:26.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.258+0000 7faafbfff640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7faae806c9f0 0x7faae806eeb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7faaec009aa0 tx=0x7faaec009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.258+0000 7faaf8ff9640 10 client.20983.objecter ms_handle_connect 0x7faae806c9f0 2024-01-27T04:20:26.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.487+0000 7faaf8ff9640 1 -- 172.21.15.110:0/3399580179 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7faaf4058ba0 con 0x7faafc1d7480 2024-01-27T04:20:26.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.487+0000 7faaf8ff9640 10 client.20983.objecter ms_dispatch 0x7faafc000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:26.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.487+0000 7faaf8ff9640 3 client.20983.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:26.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.487+0000 7faaf8ff9640 3 client.20983.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:26.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.487+0000 7faaf8ff9640 20 client.20983.objecter dump_active .. 0 homeless 2024-01-27T04:20:26.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.496+0000 7faaf8ff9640 1 -- 172.21.15.110:0/3399580179 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7faaf4054e40 con 0x7faafc1d7480 2024-01-27T04:20:26.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.496+0000 7faaf8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:26.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.496+0000 7faaf8ff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:26.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.672+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7faafc0aab10 con 0x7faae806c9f0 2024-01-27T04:20:26.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.673+0000 7faaf8ff9640 1 -- 172.21.15.110:0/3399580179 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7faafc0aab10 con 0x7faae806c9f0 2024-01-27T04:20:26.674 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 6 to deep-scrub 2024-01-27T04:20:26.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.675+0000 7fab02487640 10 librados: watch_flush enter 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.675+0000 7fab02487640 10 librados: watch_flush exit 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.675+0000 7fab02487640 20 client.20983.objecter shutdown clearing up homeless session... 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.675+0000 7fab02487640 10 client.20983.objecter successfully canceled tick 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7faae806c9f0 msgr2=0x7faae806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7faae806c9f0 0x7faae806eeb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7faaec009aa0 tx=0x7faaec009190 comp rx=0 tx=0).stop 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 10 monclient: shutdown 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faafc1d7480 msgr2=0x7faafc0ad8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faafc1d7480 0x7faafc0ad8f0 secure :-1 s=READY pgs=5101 cs=0 l=1 rev1=1 crypto rx=0x7faaf4007ec0 tx=0x7faaf400d4a0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 shutdown_connections 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7faae806c9f0 0x7faae806eeb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faafc1d7480 0x7faafc0ad8f0 unknown :-1 s=CLOSED pgs=5101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faafc142eb0 0x7faafc1d6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 --2- 172.21.15.110:0/3399580179 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faafc0b4680 0x7faafc0aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 >> 172.21.15.110:0/3399580179 conn(0x7faafc066490 msgr2=0x7faafc0b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:26.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 shutdown_connections 2024-01-27T04:20:26.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 -- 172.21.15.110:0/3399580179 wait complete. 2024-01-27T04:20:26.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.676+0000 7fab02487640 1 librados: shutdown 2024-01-27T04:20:26.689 INFO:tasks.ceph:Scrubbing osd.7 2024-01-27T04:20:26.690 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.800+0000 7f7a9215e640 10 monclient: get_monmap_and_config 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.800+0000 7f7a9215e640 10 monclient: build_initial_monmap 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.800+0000 7f7a9215e640 10 monclient: monmap: 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:26.802063+0000 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:26.802063+0000 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:26.802 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:26.803 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:26.803 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.801+0000 7f7a9215e640 1 Processor -- start 2024-01-27T04:20:26.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.801+0000 7f7a9215e640 1 -- start start 2024-01-27T04:20:26.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.801+0000 7f7a9215e640 10 monclient: init 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c06a940 0x7f7a8c1426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 10 monclient(hunting): picked mon.noname-b con 0x7f7a8c06a940 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7a8c0a9e40 0x7f7a8c142c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 10 monclient(hunting): picked mon.noname-c con 0x7f7a8c0a9e40 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a8c0b2d40 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8c0aac50 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8c0b38a0 con 0x7f7a8c0a9e40 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c06ad20 con 0x7f7a8c06a940 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a9215e640 10 monclient(hunting): authenticate will time out at 3710.156298s 2024-01-27T04:20:26.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.802+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c06a940 0x7f7a8c1426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.803+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c06a940 0x7f7a8c1426d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40860/0 (socket says 172.21.15.110:40860) 2024-01-27T04:20:26.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.803+0000 7f7a8b7fe640 1 -- 172.21.15.110:0/1572483628 learned_addr learned my addr 172.21.15.110:0/1572483628 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:26.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.803+0000 7f7a8b7fe640 10 monclient(hunting): get_auth_request con 0x7f7a8c06a940 auth_method 0 2024-01-27T04:20:26.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.803+0000 7f7a8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.803+0000 7f7a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.803+0000 7f7a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.803+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.803+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.803+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.805+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_done global_id 18944 payload 274 2024-01-27T04:20:26.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.805+0000 7f7a8b7fe640 1 -- 172.21.15.110:0/1572483628 >> v2:172.21.15.162:3300/0 conn(0x7f7a8c0a9e40 msgr2=0x7f7a8c142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:26.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.805+0000 7f7a8b7fe640 1 --2- 172.21.15.110:0/1572483628 >> v2:172.21.15.162:3300/0 conn(0x7f7a8c0a9e40 0x7f7a8c142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.805+0000 7f7a8b7fe640 1 -- 172.21.15.110:0/1572483628 >> v1:172.21.15.110:6789/0 conn(0x7f7a8c0b2d40 legacy=0x7f7a8c0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:26.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.805+0000 7f7a8b7fe640 10 monclient: _finish_hunting 0 2024-01-27T04:20:26.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.805+0000 7f7a8b7fe640 1 monclient: found mon.noname-b 2024-01-27T04:20:26.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.805+0000 7f7a8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:26.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.805+0000 7f7a8b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-01-27T04:20:26.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.805+0000 7f7a8b7fe640 1 -- 172.21.15.110:0/1572483628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c0aaf30 con 0x7f7a8c06a940 2024-01-27T04:20:26.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.807+0000 7f7a8b7fe640 1 --2- 172.21.15.110:0/1572483628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c06a940 0x7f7a8c1426d0 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009870 tx=0x7f7a7c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e31cd60367dd20f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 1 -- 172.21.15.110:0/1572483628 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c031640 con 0x7f7a8c06a940 2024-01-27T04:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 10 monclient: dump: 2024-01-27T04:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.812 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 10 monclient: _finish_auth 0 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 10 monclient: _check_auth_tickets 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 1 -- 172.21.15.110:0/1572483628 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c0317a0 con 0x7f7a8c06a940 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 1 -- 172.21.15.110:0/1572483628 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c031b30 con 0x7f7a8c06a940 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a9215e640 5 monclient: authenticate success, global_id 18944 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a9215e640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:26.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a9215e640 4 set_mon_vals no callback set 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.810+0000 7f7a8a7fc640 10 monclient: dump: 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.814 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.811+0000 7f7a9215e640 10 monclient: shutdown 2024-01-27T04:20:26.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.811+0000 7f7a9215e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:26.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.811+0000 7f7a9215e640 1 -- 172.21.15.110:0/1572483628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c06a940 msgr2=0x7f7a8c1426d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.811+0000 7f7a9215e640 1 --2- 172.21.15.110:0/1572483628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c06a940 0x7f7a8c1426d0 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009870 tx=0x7f7a7c02b8f0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.811+0000 7f7a9215e640 1 -- 172.21.15.110:0/1572483628 shutdown_connections 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.811+0000 7f7a9215e640 1 --2- 172.21.15.110:0/1572483628 >> v2:172.21.15.162:3300/0 conn(0x7f7a8c0a9e40 0x7f7a8c142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.811+0000 7f7a9215e640 1 --2- 172.21.15.110:0/1572483628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c06a940 0x7f7a8c1426d0 unknown :-1 s=CLOSED pgs=3422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.811+0000 7f7a9215e640 1 -- 172.21.15.110:0/1572483628 >> 172.21.15.110:0/1572483628 conn(0x7f7a8c0b7d70 msgr2=0x7f7a8c0b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.811+0000 7f7a9215e640 1 -- 172.21.15.110:0/1572483628 shutdown_connections 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.812+0000 7f7a9215e640 1 -- 172.21.15.110:0/1572483628 wait complete. 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.812+0000 7f7a9215e640 10 monclient: build_initial_monmap 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.812+0000 7f7a9215e640 10 monclient: monmap: 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:26.818 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.813+0000 7f7a9215e640 1 librados: starting msgr at 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.813+0000 7f7a9215e640 1 librados: starting objecter 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.813+0000 7f7a9215e640 1 Processor -- start 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.813+0000 7f7a9215e640 1 -- start start 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.813+0000 7f7a9215e640 1 librados: setting wanted keys 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.813+0000 7f7a9215e640 1 librados: calling monclient init 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.813+0000 7f7a9215e640 10 monclient: init 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a8c06a940 0x7f7a8c1314d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient(hunting): picked mon.noname-b con 0x7f7a8c06a940 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a8c0a9e40 0x7f7a8c138e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8c0a9e40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c0b2d40 0x7f7a8c13b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient(hunting): picked mon.noname-c con 0x7f7a8c0b2d40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c13aa20 con 0x7f7a8c0a9e40 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c13ac20 con 0x7f7a8c06a940 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c13af00 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.814+0000 7f7a9215e640 10 monclient(hunting): authenticate will time out at 3710.168666s 2024-01-27T04:20:26.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.815+0000 7f7a8bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c0b2d40 0x7f7a8c13b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.815+0000 7f7a8bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c0b2d40 0x7f7a8c13b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:40874/0 (socket says 172.21.15.110:40874) 2024-01-27T04:20:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.815+0000 7f7a8bfff640 1 -- 172.21.15.110:0/149671524 learned_addr learned my addr 172.21.15.110:0/149671524 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.815+0000 7f7a8bfff640 10 monclient(hunting): get_auth_request con 0x7f7a8c0b2d40 auth_method 0 2024-01-27T04:20:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.815+0000 7f7a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.815+0000 7f7a8bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.815+0000 7f7a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.816+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.816+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.816+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_done global_id 18947 payload 824 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 1 -- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a8c06a940 msgr2=0x7f7a8c1314d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a8c06a940 0x7f7a8c1314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 1 -- 172.21.15.110:0/149671524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a8c0a9e40 msgr2=0x7f7a8c138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a8c0a9e40 0x7f7a8c138e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 10 monclient: _finish_hunting 0 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 1 monclient: found mon.noname-c 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-01-27T04:20:26.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.819+0000 7f7a8bfff640 1 -- 172.21.15.110:0/149671524 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c0094d0 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.822+0000 7f7a8bfff640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c0b2d40 0x7f7a8c13b700 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400b9c0 tx=0x7f7a8400be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.822+0000 7f7a73fff640 10 client.?.objecter ms_handle_connect 0x7f7a8c0b2d40 2024-01-27T04:20:26.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.822+0000 7f7a73fff640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:26.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 1 -- 172.21.15.110:0/149671524 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8400cba0 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:26.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 10 monclient: dump: 2024-01-27T04:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.829 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 10 monclient: _finish_auth 0 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 10 monclient: _check_auth_tickets 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 1 -- 172.21.15.110:0/149671524 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8400cd00 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a9215e640 5 monclient: authenticate success, global_id 18947 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a9215e640 10 monclient: _renew_subs 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a9215e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8c138570 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:26.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 1 -- 172.21.15.110:0/149671524 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84014680 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a73fff640 10 monclient: dump: 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:26.831 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a89ffb640 4 set_mon_vals no callback set 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a9215e640 10 client.18947.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a9215e640 10 monclient: _renew_subs 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a9215e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8c13bc40 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.826+0000 7f7a9215e640 1 librados: init done 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.827+0000 7f7a9215e640 10 client.18947.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:26.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.827+0000 7f7a9215e640 15 client.18947.objecter _session_command_op_assign -1 1 2024-01-27T04:20:26.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.827+0000 7f7a9215e640 20 client.18947.objecter _get_session osd=-1 returning homeless 2024-01-27T04:20:26.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.827+0000 7f7a9215e640 10 client.18947.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:26.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.827+0000 7f7a9215e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-01-27T04:20:26.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.827+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7a8c0b4590 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.848+0000 7f7a73fff640 1 -- 172.21.15.110:0/149671524 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f7a8401d3b0 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.848+0000 7f7a73fff640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7a7806c9f0 0x7f7a7806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.848+0000 7f7a8b7fe640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7a7806c9f0 0x7f7a7806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.848+0000 7f7a8b7fe640 10 monclient: get_auth_request con 0x7f7a7806c9f0 auth_method 0 2024-01-27T04:20:26.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.849+0000 7f7a8b7fe640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7a7806c9f0 0x7f7a7806eeb0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009870 tx=0x7f7a7c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.849+0000 7f7a73fff640 10 client.18947.objecter ms_handle_connect 0x7f7a7806c9f0 2024-01-27T04:20:26.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 1 -- 172.21.15.110:0/149671524 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f7a84059130 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 10 client.18947.objecter ms_dispatch 0x7f7a8c000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:26.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 3 client.18947.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:26.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 3 client.18947.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:26.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 10 client.18947.objecter checking command 1 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7f7a78074770 0x7f7a78076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 20 client.18947.objecter _get_session s=0x7f7a780740a0 osd=7 3 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 20 client.18947.objecter put_session s=0x7f7a780740a0 osd=7 3 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 15 client.18947.objecter _session_command_op_remove -1 1 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 20 client.18947.objecter _get_session s=0x7f7a780740a0 osd=7 3 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 20 client.18947.objecter get_session s=0x7f7a780740a0 osd=7 3 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 15 client.18947.objecter _session_command_op_assign 7 1 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 20 client.18947.objecter put_session s=0x7f7a780740a0 osd=7 4 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 10 client.18947.objecter _send_command 1 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 1 -- 172.21.15.110:0/149671524 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a780772e0 con 0x7f7a78074770 2024-01-27T04:20:26.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.852+0000 7f7a73fff640 20 client.18947.objecter dump_active .. 0 homeless 2024-01-27T04:20:26.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.853+0000 7f7a8affd640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7f7a78074770 0x7f7a78076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:26.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.853+0000 7f7a8affd640 10 monclient: get_auth_request con 0x7f7a78074770 auth_method 0 2024-01-27T04:20:26.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.856+0000 7f7a73fff640 10 client.18947.objecter ms_handle_connect 0x7f7a78074770 2024-01-27T04:20:26.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.856+0000 7f7a8affd640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7f7a78074770 0x7f7a78076c10 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:26.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.882+0000 7f7a73fff640 1 -- 172.21.15.110:0/149671524 <== osd.7 v2:172.21.15.162:6818/68785612 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7a780772e0 con 0x7f7a78074770 2024-01-27T04:20:26.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.882+0000 7f7a73fff640 10 client.18947.objecter ms_dispatch 0x7f7a8c000b90 command_reply(tid 1: 0 ) v1 2024-01-27T04:20:26.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.882+0000 7f7a73fff640 10 client.18947.objecter _finish_command 1 = system:0 2024-01-27T04:20:26.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.882+0000 7f7a73fff640 20 client.18947.objecter put_session s=0x7f7a780740a0 osd=7 4 2024-01-27T04:20:26.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.882+0000 7f7a73fff640 15 client.18947.objecter _session_command_op_remove 7 1 2024-01-27T04:20:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 10 client.18947.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-27T04:20:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 15 client.18947.objecter _session_command_op_assign -1 2 2024-01-27T04:20:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 20 client.18947.objecter _get_session s=0x7f7a780740a0 osd=7 3 2024-01-27T04:20:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 20 client.18947.objecter put_session s=0x7f7a780740a0 osd=7 3 2024-01-27T04:20:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 20 client.18947.objecter _get_session s=0x7f7a780740a0 osd=7 3 2024-01-27T04:20:26.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 15 client.18947.objecter _session_command_op_remove -1 2 2024-01-27T04:20:26.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 20 client.18947.objecter get_session s=0x7f7a780740a0 osd=7 3 2024-01-27T04:20:26.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 15 client.18947.objecter _session_command_op_assign 7 2 2024-01-27T04:20:26.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 20 client.18947.objecter put_session s=0x7f7a780740a0 osd=7 4 2024-01-27T04:20:26.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 10 client.18947.objecter _send_command 2 2024-01-27T04:20:26.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.907+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 --> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7a8c13b420 con 0x7f7a78074770 2024-01-27T04:20:26.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.916+0000 7f7a73fff640 1 -- 172.21.15.110:0/149671524 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=1334) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a8400ce70 con 0x7f7a8c0b2d40 2024-01-27T04:20:26.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.916+0000 7f7a73fff640 10 monclient: handle_get_version_reply finishing 1 version 1334 2024-01-27T04:20:26.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a73fff640 1 -- 172.21.15.110:0/149671524 <== osd.7 v2:172.21.15.162:6818/68785612 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7a8c13b420 con 0x7f7a78074770 2024-01-27T04:20:26.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a73fff640 10 client.18947.objecter ms_dispatch 0x7f7a8c000b90 command_reply(tid 2: 0 ) v1 2024-01-27T04:20:26.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a73fff640 10 client.18947.objecter _finish_command 2 = system:0 2024-01-27T04:20:26.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a73fff640 20 client.18947.objecter put_session s=0x7f7a780740a0 osd=7 4 2024-01-27T04:20:26.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a73fff640 15 client.18947.objecter _session_command_op_remove 7 2 2024-01-27T04:20:26.924 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-01-27T04:20:26.924 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-27T04:20:26.924 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-01-27T04:20:26.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 10 librados: watch_flush enter 2024-01-27T04:20:26.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 10 librados: watch_flush exit 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 10 client.18947.objecter close_session for osd.7 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7f7a78074770 msgr2=0x7f7a78076c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6818/68785612,v1:172.21.15.162:6819/68785612] conn(0x7f7a78074770 0x7f7a78076c10 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 20 client.18947.objecter put_session s=0x7f7a780740a0 osd=7 1 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 20 client.18947.objecter shutdown clearing up homeless session... 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 10 client.18947.objecter successfully canceled tick 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7a7806c9f0 msgr2=0x7f7a7806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f7a7806c9f0 0x7f7a7806eeb0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009870 tx=0x7f7a7c030000 comp rx=0 tx=0).stop 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 10 monclient: shutdown 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c0b2d40 msgr2=0x7f7a8c13b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:26.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 --2- 172.21.15.110:0/149671524 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a8c0b2d40 0x7f7a8c13b700 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400b9c0 tx=0x7f7a8400be80 comp rx=0 tx=0).stop 2024-01-27T04:20:26.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a8bfff640 1 -- 172.21.15.110:0/149671524 reap_dead start 2024-01-27T04:20:26.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 shutdown_connections 2024-01-27T04:20:26.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 >> 172.21.15.110:0/149671524 conn(0x7f7a8c0b7d70 msgr2=0x7f7a8c1426d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:26.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 shutdown_connections 2024-01-27T04:20:26.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 -- 172.21.15.110:0/149671524 wait complete. 2024-01-27T04:20:26.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:26.922+0000 7f7a9215e640 1 librados: shutdown 2024-01-27T04:20:26.934 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-27T04:20:27.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.035+0000 7ff361e7c640 10 monclient: get_monmap_and_config 2024-01-27T04:20:27.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.035+0000 7ff361e7c640 10 monclient: build_initial_monmap 2024-01-27T04:20:27.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.035+0000 7ff361e7c640 10 monclient: monmap: 2024-01-27T04:20:27.036 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:27.036 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.036 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:27.036271+0000 2024-01-27T04:20:27.037 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:27.036271+0000 2024-01-27T04:20:27.037 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:27.037 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.037 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:27.037 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:27.037 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:27.037 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:27.037 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:27.037 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.035+0000 7ff361e7c640 1 Processor -- start 2024-01-27T04:20:27.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.035+0000 7ff361e7c640 1 -- start start 2024-01-27T04:20:27.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.035+0000 7ff361e7c640 10 monclient: init 2024-01-27T04:20:27.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:27.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-01-27T04:20:27.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff35c0b4b00 0x7ff35c0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 10 monclient(hunting): picked mon.noname-c con 0x7ff35c0b4b00 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff35c13f020 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff35c13f490 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0aaf30 con 0x7ff35c13f020 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0ab210 con 0x7ff35c13f490 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c0b44d0 con 0x7ff35c0b4b00 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.036+0000 7ff361e7c640 10 monclient(hunting): authenticate will time out at 3710.390560s 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.037+0000 7ff35b7fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff35c0b4b00 0x7ff35c0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.037+0000 7ff352ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff35c13f020 0x7ff35c142060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48106/0 (socket says 172.21.15.110:48106) 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.037+0000 7ff352ffd640 1 -- 172.21.15.110:0/550969667 learned_addr learned my addr 172.21.15.110:0/550969667 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T04:20:27.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.037+0000 7ff35b7fe640 10 monclient(hunting): get_auth_request con 0x7ff35c0b4b00 auth_method 0 2024-01-27T04:20:27.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.037+0000 7ff35b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:27.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.037+0000 7ff35b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.037+0000 7ff35b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.038+0000 7ff35b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:27.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.038+0000 7ff35b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:27.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.038+0000 7ff35b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:27.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.039+0000 7ff35affd640 1 -- 172.21.15.110:0/550969667 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922792732 0 0) 0x7ff35c0aaf30 con 0x7ff35c13f020 2024-01-27T04:20:27.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.039+0000 7ff35affd640 10 monclient(hunting): _negotiate 2024-01-27T04:20:27.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.039+0000 7ff35affd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.039+0000 7ff35affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.039+0000 7ff35affd640 10 monclient(hunting): my global_id is 17079 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.039+0000 7ff35affd640 1 -- 172.21.15.110:0/550969667 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff348003520 con 0x7ff35c13f020 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.040+0000 7ff35b7fe640 10 monclient(hunting): handle_auth_done global_id 20986 payload 274 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.040+0000 7ff35b7fe640 1 -- 172.21.15.110:0/550969667 >> v1:172.21.15.162:6789/0 conn(0x7ff35c13f490 legacy=0x7ff35c142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.040+0000 7ff35b7fe640 1 -- 172.21.15.110:0/550969667 >> v1:172.21.15.110:6789/0 conn(0x7ff35c13f020 legacy=0x7ff35c142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.040+0000 7ff35b7fe640 10 monclient: _finish_hunting 0 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.040+0000 7ff35b7fe640 1 monclient: found mon.noname-c 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.040+0000 7ff35b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.040+0000 7ff35b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T04:20:27.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.040+0000 7ff35b7fe640 1 -- 172.21.15.110:0/550969667 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c13f900 con 0x7ff35c0b4b00 2024-01-27T04:20:27.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.045+0000 7ff35b7fe640 1 --2- 172.21.15.110:0/550969667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c0b4b00 0x7ff35c0b3db0 secure :-1 s=READY pgs=5103 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0097a0 tx=0x7ff34c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=441942e51c89d1a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:27.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 1 -- 172.21.15.110:0/550969667 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c02e070 con 0x7ff35c0b4b00 2024-01-27T04:20:27.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:27.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:27.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 10 monclient: dump: 2024-01-27T04:20:27.050 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 10 monclient: _finish_auth 0 2024-01-27T04:20:27.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 10 monclient: _check_auth_tickets 2024-01-27T04:20:27.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:27.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 1 -- 172.21.15.110:0/550969667 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c0315a0 con 0x7ff35c0b4b00 2024-01-27T04:20:27.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 5 monclient: authenticate success, global_id 20986 2024-01-27T04:20:27.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:27.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:27.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 1 -- 172.21.15.110:0/550969667 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c031930 con 0x7ff35c0b4b00 2024-01-27T04:20:27.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:27.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 4 set_mon_vals no callback set 2024-01-27T04:20:27.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:27.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:27.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff35affd640 10 monclient: dump: 2024-01-27T04:20:27.053 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:27.053 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.053 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.053 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.053 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:27.053 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.053 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:27.054 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:27.054 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:27.054 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 10 monclient: shutdown 2024-01-27T04:20:27.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:27.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 1 -- 172.21.15.110:0/550969667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c0b4b00 msgr2=0x7ff35c0b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 1 --2- 172.21.15.110:0/550969667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c0b4b00 0x7ff35c0b3db0 secure :-1 s=READY pgs=5103 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0097a0 tx=0x7ff34c02b840 comp rx=0 tx=0).stop 2024-01-27T04:20:27.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 1 -- 172.21.15.110:0/550969667 shutdown_connections 2024-01-27T04:20:27.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 1 --2- 172.21.15.110:0/550969667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c0b4b00 0x7ff35c0b3db0 unknown :-1 s=CLOSED pgs=5103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.049+0000 7ff361e7c640 1 -- 172.21.15.110:0/550969667 >> 172.21.15.110:0/550969667 conn(0x7ff35c066490 msgr2=0x7ff35c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:27.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.050+0000 7ff361e7c640 1 -- 172.21.15.110:0/550969667 shutdown_connections 2024-01-27T04:20:27.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.050+0000 7ff361e7c640 1 -- 172.21.15.110:0/550969667 wait complete. 2024-01-27T04:20:27.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.050+0000 7ff361e7c640 10 monclient: build_initial_monmap 2024-01-27T04:20:27.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.050+0000 7ff361e7c640 10 monclient: monmap: 2024-01-27T04:20:27.059 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:27.059 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:27.059 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:27.059 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:27.059 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:27.060 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.060 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:27.060 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:27.060 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:27.060 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.050+0000 7ff361e7c640 1 librados: starting msgr at 2024-01-27T04:20:27.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.050+0000 7ff361e7c640 1 librados: starting objecter 2024-01-27T04:20:27.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 Processor -- start 2024-01-27T04:20:27.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 -- start start 2024-01-27T04:20:27.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 librados: setting wanted keys 2024-01-27T04:20:27.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 librados: calling monclient init 2024-01-27T04:20:27.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient: init 2024-01-27T04:20:27.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:27.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T04:20:27.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0b4b00 0x7ff35c0abc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient(hunting): picked mon.noname-a con 0x7ff35c0b4b00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:27.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c13f020 0x7ff35c0ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient(hunting): picked mon.noname-b con 0x7ff35c13f020 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:27.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c13f490 0x7ff35c13b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient(hunting): picked mon.noname-c con 0x7ff35c13f490 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:27.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:27.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0adb90 con 0x7ff35c0b4b00 2024-01-27T04:20:27.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:27.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0acf40 con 0x7ff35c13f020 2024-01-27T04:20:27.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:27.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0adde0 con 0x7ff35c13f490 2024-01-27T04:20:27.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:27.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.051+0000 7ff361e7c640 10 monclient(hunting): authenticate will time out at 3710.405814s 2024-01-27T04:20:27.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0b4b00 0x7ff35c0abc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff352ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c13f020 0x7ff35c0ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c13f490 0x7ff35c13b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0b4b00 0x7ff35c0abc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45070/0 (socket says 172.21.15.110:45070) 2024-01-27T04:20:27.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35b7fe640 1 -- 172.21.15.110:0/519722241 learned_addr learned my addr 172.21.15.110:0/519722241 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:27.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35b7fe640 10 monclient(hunting): get_auth_request con 0x7ff35c0b4b00 auth_method 0 2024-01-27T04:20:27.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:27.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff352ffd640 10 monclient(hunting): get_auth_request con 0x7ff35c13f020 auth_method 0 2024-01-27T04:20:27.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff352ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:27.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff352ffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff352ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35bfff640 10 monclient(hunting): get_auth_request con 0x7ff35c13f490 auth_method 0 2024-01-27T04:20:27.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:27.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.052+0000 7ff35bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.053+0000 7ff352ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:27.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.053+0000 7ff352ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:27.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.053+0000 7ff352ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:27.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.053+0000 7ff35b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:27.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.053+0000 7ff35b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:27.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.053+0000 7ff35b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:27.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.053+0000 7ff35bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:27.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.053+0000 7ff35bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:27.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.053+0000 7ff35bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:27.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 10 monclient(hunting): handle_auth_done global_id 20992 payload 824 2024-01-27T04:20:27.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 1 -- 172.21.15.110:0/519722241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c13f490 msgr2=0x7ff35c13b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c13f490 0x7ff35c13b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 1 -- 172.21.15.110:0/519722241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0b4b00 msgr2=0x7ff35c0abc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0b4b00 0x7ff35c0abc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 10 monclient: _finish_hunting 0 2024-01-27T04:20:27.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 1 monclient: found mon.noname-b 2024-01-27T04:20:27.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:27.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:27.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.055+0000 7ff352ffd640 1 -- 172.21.15.110:0/519722241 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c009400 con 0x7ff35c13f020 2024-01-27T04:20:27.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.056+0000 7ff35b7fe640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0b4b00 0x7ff35c0abc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-27T04:20:27.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.058+0000 7ff352ffd640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c13f020 0x7ff35c0ac390 secure :-1 s=READY pgs=5104 cs=0 l=1 rev1=1 crypto rx=0x7ff34400c430 tx=0x7ff34400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:27.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.058+0000 7ff358ff9640 10 client.?.objecter ms_handle_connect 0x7ff35c13f020 2024-01-27T04:20:27.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.058+0000 7ff358ff9640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:27.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 1 -- 172.21.15.110:0/519722241 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344005ce0 con 0x7ff35c13f020 2024-01-27T04:20:27.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:27.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:27.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 10 monclient: dump: 2024-01-27T04:20:27.072 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:27.072 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.072 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.072 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.072 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:27.072 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.072 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:27.073 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:27.073 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:27.073 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 10 monclient: _finish_auth 0 2024-01-27T04:20:27.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 10 monclient: _check_auth_tickets 2024-01-27T04:20:27.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:27.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 1 -- 172.21.15.110:0/519722241 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff344005e40 con 0x7ff35c13f020 2024-01-27T04:20:27.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:27.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 1 -- 172.21.15.110:0/519722241 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3440072d0 con 0x7ff35c13f020 2024-01-27T04:20:27.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:27.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:27.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff358ff9640 10 monclient: dump: 2024-01-27T04:20:27.074 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:27.074 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.075 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.075 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.075 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:27.075 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.075 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:27.075 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:27.076 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:27.076 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff35a7fc640 4 set_mon_vals no callback set 2024-01-27T04:20:27.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff361e7c640 5 monclient: authenticate success, global_id 20992 2024-01-27T04:20:27.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff361e7c640 10 monclient: _renew_subs 2024-01-27T04:20:27.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff361e7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:27.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35c0ae0c0 con 0x7ff35c13f020 2024-01-27T04:20:27.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff361e7c640 10 client.20992.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:27.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff361e7c640 10 monclient: _renew_subs 2024-01-27T04:20:27.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff361e7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:27.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c13be90 con 0x7ff35c13f020 2024-01-27T04:20:27.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.061+0000 7ff361e7c640 1 librados: init done 2024-01-27T04:20:27.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.062+0000 7ff361e7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:27.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.062+0000 7ff361e7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:27.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.062+0000 7ff361e7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:27.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.062+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35c0acc60 con 0x7ff35c13f020 2024-01-27T04:20:27.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.080+0000 7ff358ff9640 1 -- 172.21.15.110:0/519722241 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7ff344007430 con 0x7ff35c13f020 2024-01-27T04:20:27.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.081+0000 7ff358ff9640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff33c06c9f0 0x7ff33c06eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.081+0000 7ff35b7fe640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff33c06c9f0 0x7ff33c06eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.081+0000 7ff35b7fe640 10 monclient: get_auth_request con 0x7ff33c06c9f0 auth_method 0 2024-01-27T04:20:27.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.081+0000 7ff35b7fe640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff33c06c9f0 0x7ff33c06eeb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff34c000c00 tx=0x7ff34c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:27.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.081+0000 7ff358ff9640 10 client.20992.objecter ms_handle_connect 0x7ff33c06c9f0 2024-01-27T04:20:27.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.084+0000 7ff358ff9640 1 -- 172.21.15.110:0/519722241 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7ff344057c80 con 0x7ff35c13f020 2024-01-27T04:20:27.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.084+0000 7ff358ff9640 10 client.20992.objecter ms_dispatch 0x7ff35c000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:27.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.084+0000 7ff358ff9640 3 client.20992.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:27.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.084+0000 7ff358ff9640 3 client.20992.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:27.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.085+0000 7ff358ff9640 20 client.20992.objecter dump_active .. 0 homeless 2024-01-27T04:20:27.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.323+0000 7ff358ff9640 1 -- 172.21.15.110:0/519722241 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7ff344053f20 con 0x7ff35c13f020 2024-01-27T04:20:27.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.323+0000 7ff358ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:27.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.323+0000 7ff358ff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:27.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.495+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7ff35c0b4ee0 con 0x7ff33c06c9f0 2024-01-27T04:20:27.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.496+0000 7ff358ff9640 1 -- 172.21.15.110:0/519722241 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff34c031b30 con 0x7ff33c06c9f0 2024-01-27T04:20:27.497 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 7 to deep-scrub 2024-01-27T04:20:27.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.497+0000 7ff361e7c640 10 librados: watch_flush enter 2024-01-27T04:20:27.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 10 librados: watch_flush exit 2024-01-27T04:20:27.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 20 client.20992.objecter shutdown clearing up homeless session... 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 10 client.20992.objecter successfully canceled tick 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff33c06c9f0 msgr2=0x7ff33c06eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff33c06c9f0 0x7ff33c06eeb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff34c000c00 tx=0x7ff34c03d040 comp rx=0 tx=0).stop 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 10 monclient: shutdown 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c13f020 msgr2=0x7ff35c0ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c13f020 0x7ff35c0ac390 secure :-1 s=READY pgs=5104 cs=0 l=1 rev1=1 crypto rx=0x7ff34400c430 tx=0x7ff34400c8f0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 shutdown_connections 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7ff33c06c9f0 0x7ff33c06eeb0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff35c13f490 0x7ff35c13b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff35c13f020 0x7ff35c0ac390 unknown :-1 s=CLOSED pgs=5104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 --2- 172.21.15.110:0/519722241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff35c0b4b00 0x7ff35c0abc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 >> 172.21.15.110:0/519722241 conn(0x7ff35c066490 msgr2=0x7ff35c0b5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 shutdown_connections 2024-01-27T04:20:27.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 -- 172.21.15.110:0/519722241 wait complete. 2024-01-27T04:20:27.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.498+0000 7ff361e7c640 1 librados: shutdown 2024-01-27T04:20:27.510 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T04:20:27.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.609+0000 7f12e9b89640 10 monclient: get_monmap_and_config 2024-01-27T04:20:27.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.609+0000 7f12e9b89640 10 monclient: build_initial_monmap 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.609+0000 7f12e9b89640 10 monclient: monmap: 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:27.611109+0000 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:27.611109+0000 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.610+0000 7f12e9b89640 1 Processor -- start 2024-01-27T04:20:27.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.610+0000 7f12e9b89640 1 -- start start 2024-01-27T04:20:27.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.610+0000 7f12e9b89640 10 monclient: init 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12e406bcb0 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12e406c090 addr v1:172.21.15.110:6789/0 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f12e4145eb0 0x7f12e414d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 10 monclient(hunting): picked mon.noname-c con 0x7f12e4145eb0 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e40ae4b0 con 0x7f12e406c090 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e40ae090 con 0x7f12e406bcb0 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e406ca50 con 0x7f12e4145eb0 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e9b89640 10 monclient(hunting): authenticate will time out at 3710.965285s 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e2ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f12e406c090 0x7f12e41457a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48118/0 (socket says 172.21.15.110:48118) 2024-01-27T04:20:27.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.611+0000 7f12e2ffd640 1 -- 172.21.15.110:0/3748387333 learned_addr learned my addr 172.21.15.110:0/3748387333 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.612+0000 7f12e3fff640 1 --2- 172.21.15.110:0/3748387333 >> v2:172.21.15.162:3300/0 conn(0x7f12e4145eb0 0x7f12e414d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.612+0000 7f12e3fff640 10 monclient(hunting): get_auth_request con 0x7f12e4145eb0 auth_method 0 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.612+0000 7f12e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.612+0000 7f12e3fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.612+0000 7f12e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.613+0000 7f12e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.613+0000 7f12e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.613+0000 7f12e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.614+0000 7f12e27fc640 1 -- 172.21.15.110:0/3748387333 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117874502 0 0) 0x7f12e40ae4b0 con 0x7f12e406c090 2024-01-27T04:20:27.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.614+0000 7f12e27fc640 10 monclient(hunting): _negotiate 2024-01-27T04:20:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.614+0000 7f12e27fc640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.614+0000 7f12e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.614+0000 7f12e27fc640 10 monclient(hunting): my global_id is 17085 2024-01-27T04:20:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.614+0000 7f12e27fc640 1 -- 172.21.15.110:0/3748387333 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12cc003520 con 0x7f12e406c090 2024-01-27T04:20:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.615+0000 7f12e3fff640 10 monclient(hunting): handle_auth_done global_id 20995 payload 274 2024-01-27T04:20:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.615+0000 7f12e3fff640 1 -- 172.21.15.110:0/3748387333 >> v1:172.21.15.162:6789/0 conn(0x7f12e406bcb0 legacy=0x7f12e413f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.615+0000 7f12e3fff640 1 -- 172.21.15.110:0/3748387333 >> v1:172.21.15.110:6789/0 conn(0x7f12e406c090 legacy=0x7f12e41457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.615+0000 7f12e3fff640 10 monclient: _finish_hunting 0 2024-01-27T04:20:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.615+0000 7f12e3fff640 1 monclient: found mon.noname-c 2024-01-27T04:20:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.615+0000 7f12e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.615+0000 7f12e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T04:20:27.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.615+0000 7f12e3fff640 1 -- 172.21.15.110:0/3748387333 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e4146320 con 0x7f12e4145eb0 2024-01-27T04:20:27.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.618+0000 7f12e3fff640 1 --2- 172.21.15.110:0/3748387333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12e4145eb0 0x7f12e414d8a0 secure :-1 s=READY pgs=5106 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009d90 tx=0x7f12dc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=294214afbdb076d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 1 -- 172.21.15.110:0/3748387333 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc02cc60 con 0x7f12e4145eb0 2024-01-27T04:20:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:27.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 10 monclient: dump: 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 10 monclient: _finish_auth 0 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 10 monclient: _check_auth_tickets 2024-01-27T04:20:27.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 1 -- 172.21.15.110:0/3748387333 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc02cdc0 con 0x7f12e4145eb0 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 1 -- 172.21.15.110:0/3748387333 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc034660 con 0x7f12e4145eb0 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e27fc640 10 monclient: dump: 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:27.625 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:27.626 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e9b89640 5 monclient: authenticate success, global_id 20995 2024-01-27T04:20:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e9b89640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:27.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.622+0000 7f12e9b89640 4 set_mon_vals no callback set 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 10 monclient: shutdown 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 1 -- 172.21.15.110:0/3748387333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12e4145eb0 msgr2=0x7f12e414d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 1 --2- 172.21.15.110:0/3748387333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12e4145eb0 0x7f12e414d8a0 secure :-1 s=READY pgs=5106 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009d90 tx=0x7f12dc02beb0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 1 -- 172.21.15.110:0/3748387333 shutdown_connections 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 1 --2- 172.21.15.110:0/3748387333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12e4145eb0 0x7f12e414d8a0 unknown :-1 s=CLOSED pgs=5106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 1 -- 172.21.15.110:0/3748387333 >> 172.21.15.110:0/3748387333 conn(0x7f12e40b2640 msgr2=0x7f12e40b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 1 -- 172.21.15.110:0/3748387333 shutdown_connections 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 1 -- 172.21.15.110:0/3748387333 wait complete. 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 10 monclient: build_initial_monmap 2024-01-27T04:20:27.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.623+0000 7f12e9b89640 10 monclient: monmap: 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 1 librados: starting msgr at 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 1 librados: starting objecter 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 1 Processor -- start 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 1 -- start start 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 1 librados: setting wanted keys 2024-01-27T04:20:27.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 1 librados: calling monclient init 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 10 monclient: init 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.624+0000 7f12e9b89640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12e406bcb0 0x7f12e40b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 10 monclient(hunting): picked mon.noname-a con 0x7f12e406bcb0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12e406c090 0x7f12e4136490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 10 monclient(hunting): picked mon.noname-b con 0x7f12e406c090 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12e4145eb0 0x7f12e41435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 10 monclient(hunting): picked mon.noname-c con 0x7f12e4145eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e40b5360 con 0x7f12e406bcb0 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e4145290 con 0x7f12e406c090 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:27.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e4144db0 con 0x7f12e4145eb0 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e9b89640 10 monclient(hunting): authenticate will time out at 3710.979025s 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12e406bcb0 0x7f12e40b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12e4145eb0 0x7f12e41435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12e406c090 0x7f12e4136490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12e406bcb0 0x7f12e40b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45080/0 (socket says 172.21.15.110:45080) 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e37fe640 1 -- 172.21.15.110:0/2624135921 learned_addr learned my addr 172.21.15.110:0/2624135921 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e37fe640 10 monclient(hunting): get_auth_request con 0x7f12e406bcb0 auth_method 0 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e37fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.625+0000 7f12e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e3fff640 10 monclient(hunting): get_auth_request con 0x7f12e4145eb0 auth_method 0 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e3fff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e2ffd640 10 monclient(hunting): get_auth_request con 0x7f12e406c090 auth_method 0 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:27.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.626+0000 7f12e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 10 monclient(hunting): handle_auth_done global_id 17088 payload 824 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 1 -- 172.21.15.110:0/2624135921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12e4145eb0 msgr2=0x7f12e41435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12e4145eb0 0x7f12e41435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 1 -- 172.21.15.110:0/2624135921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12e406c090 msgr2=0x7f12e4136490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12e406c090 0x7f12e4136490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 10 monclient: _finish_hunting 0 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 1 monclient: found mon.noname-a 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.629+0000 7f12e37fe640 1 -- 172.21.15.110:0/2624135921 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc009a70 con 0x7f12e406bcb0 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.631+0000 7f12e37fe640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12e406bcb0 0x7f12e40b3730 secure :-1 s=READY pgs=3475 cs=0 l=1 rev1=1 crypto rx=0x7f12d400c400 tx=0x7f12d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.631+0000 7f12cbfff640 10 client.?.objecter ms_handle_connect 0x7f12e406bcb0 2024-01-27T04:20:27.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.631+0000 7f12cbfff640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12cbfff640 1 -- 172.21.15.110:0/2624135921 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d40052d0 con 0x7f12e406bcb0 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12cbfff640 10 monclient: dump: 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12cbfff640 10 monclient: _finish_auth 0 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12cbfff640 10 monclient: _check_auth_tickets 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12e9b89640 5 monclient: authenticate success, global_id 17088 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12e9b89640 10 monclient: _renew_subs 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12e9b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e4145030 con 0x7f12e406bcb0 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12e9b89640 10 client.17088.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12e9b89640 10 monclient: _renew_subs 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12e9b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e4143cc0 con 0x7f12e406bcb0 2024-01-27T04:20:27.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.633+0000 7f12e9b89640 1 librados: init done 2024-01-27T04:20:27.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.634+0000 7f12e9b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:27.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.634+0000 7f12e9b89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:27.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.634+0000 7f12e9b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-01-27T04:20:27.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.634+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e404ea50 con 0x7f12e406bcb0 2024-01-27T04:20:27.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.635+0000 7f12cbfff640 1 -- 172.21.15.110:0/2624135921 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d4005430 con 0x7f12e406bcb0 2024-01-27T04:20:27.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.635+0000 7f12cbfff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:27.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.635+0000 7f12cbfff640 1 -- 172.21.15.110:0/2624135921 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d4007240 con 0x7f12e406bcb0 2024-01-27T04:20:27.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.635+0000 7f12cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:27.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.635+0000 7f12cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.635+0000 7f12cbfff640 10 monclient: dump: 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:27.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.635+0000 7f12e1ffb640 4 set_mon_vals no callback set 2024-01-27T04:20:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.882+0000 7f12cbfff640 1 -- 172.21.15.110:0/2624135921 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f12d40073a0 con 0x7f12e406bcb0 2024-01-27T04:20:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.882+0000 7f12cbfff640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f12d006c9a0 0x7f12d006ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.883+0000 7f12e2ffd640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f12d006c9a0 0x7f12d006ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.883+0000 7f12e2ffd640 10 monclient: get_auth_request con 0x7f12d006c9a0 auth_method 0 2024-01-27T04:20:27.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.883+0000 7f12e2ffd640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f12d006c9a0 0x7f12d006ee60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f12e40b5780 tx=0x7f12d8000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:27.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.883+0000 7f12cbfff640 10 client.17088.objecter ms_handle_connect 0x7f12d006c9a0 2024-01-27T04:20:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.895+0000 7f12cbfff640 1 -- 172.21.15.110:0/2624135921 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f12d4007650 con 0x7f12e406bcb0 2024-01-27T04:20:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.895+0000 7f12cbfff640 10 client.17088.objecter ms_dispatch 0x7f12e4000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.895+0000 7f12cbfff640 3 client.17088.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.895+0000 7f12cbfff640 3 client.17088.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.895+0000 7f12cbfff640 20 client.17088.objecter dump_active .. 0 homeless 2024-01-27T04:20:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.895+0000 7f12cbfff640 1 -- 172.21.15.110:0/2624135921 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f12d4053f00 con 0x7f12e406bcb0 2024-01-27T04:20:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.895+0000 7f12cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:27.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:27.895+0000 7f12cbfff640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:28.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.086+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12e41432c0 con 0x7f12d006c9a0 2024-01-27T04:20:28.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.090+0000 7f12cbfff640 1 -- 172.21.15.110:0/2624135921 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150793 (secure 0 0 0) 0x7f12e41432c0 con 0x7f12d006c9a0 2024-01-27T04:20:28.093 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T04:20:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 10 librados: watch_flush enter 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 10 librados: watch_flush exit 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 20 client.17088.objecter shutdown clearing up homeless session... 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 10 client.17088.objecter successfully canceled tick 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f12d006c9a0 msgr2=0x7f12d006ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f12d006c9a0 0x7f12d006ee60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f12e40b5780 tx=0x7f12d8000bc0 comp rx=0 tx=0).stop 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 10 monclient: shutdown 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12e406bcb0 msgr2=0x7f12e40b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12e406bcb0 0x7f12e40b3730 secure :-1 s=READY pgs=3475 cs=0 l=1 rev1=1 crypto rx=0x7f12d400c400 tx=0x7f12d400c8c0 comp rx=0 tx=0).stop 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 shutdown_connections 2024-01-27T04:20:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f12d006c9a0 0x7f12d006ee60 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12e4145eb0 0x7f12e41435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12e406c090 0x7f12e4136490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 --2- 172.21.15.110:0/2624135921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12e406bcb0 0x7f12e40b3730 unknown :-1 s=CLOSED pgs=3475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 >> 172.21.15.110:0/2624135921 conn(0x7f12e40b2640 msgr2=0x7f12e406e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 shutdown_connections 2024-01-27T04:20:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 -- 172.21.15.110:0/2624135921 wait complete. 2024-01-27T04:20:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:28.093+0000 7f12e9b89640 1 librados: shutdown 2024-01-27T04:20:28.106 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":2456,"stamp":"2024-01-27T04:20:27.555067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":578260,"kb_used_data":40056,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754396460,"statfs":{"total":773094113280,"available":772501975040,"internally_reserved":0,"allocated":41017344,"data_stored":19217842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.914931"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2558,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.714798+0000","last_change":"2024-01-27T03:45:12.104231+0000","last_active":"2024-01-27T04:20:16.714798+0000","last_peered":"2024-01-27T04:20:16.714798+0000","last_clean":"2024-01-27T04:20:16.714798+0000","last_became_active":"2024-01-27T03:45:12.095646+0000","last_became_peered":"2024-01-27T03:45:12.095646+0000","last_unstale":"2024-01-27T04:20:16.714798+0000","last_undegraded":"2024-01-27T04:20:16.714798+0000","last_fullsized":"2024-01-27T04:20:16.714798+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2570,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.848121+0000","last_change":"2024-01-27T03:45:50.347781+0000","last_active":"2024-01-27T04:20:16.848121+0000","last_peered":"2024-01-27T04:20:16.848121+0000","last_clean":"2024-01-27T04:20:16.848121+0000","last_became_active":"2024-01-27T03:45:50.339311+0000","last_became_peered":"2024-01-27T03:45:50.339311+0000","last_unstale":"2024-01-27T04:20:16.848121+0000","last_undegraded":"2024-01-27T04:20:16.848121+0000","last_fullsized":"2024-01-27T04:20:16.848121+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2563,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.658924+0000","last_change":"2024-01-27T03:45:01.242765+0000","last_active":"2024-01-27T04:20:16.658924+0000","last_peered":"2024-01-27T04:20:16.658924+0000","last_clean":"2024-01-27T04:20:16.658924+0000","last_became_active":"2024-01-27T03:45:01.173317+0000","last_became_peered":"2024-01-27T03:45:01.173317+0000","last_unstale":"2024-01-27T04:20:16.658924+0000","last_undegraded":"2024-01-27T04:20:16.658924+0000","last_fullsized":"2024-01-27T04:20:16.658924+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2553,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.849854+0000","last_change":"2024-01-27T03:45:08.326818+0000","last_active":"2024-01-27T04:20:16.849854+0000","last_peered":"2024-01-27T04:20:16.849854+0000","last_clean":"2024-01-27T04:20:16.849854+0000","last_became_active":"2024-01-27T03:45:08.187722+0000","last_became_peered":"2024-01-27T03:45:08.187722+0000","last_unstale":"2024-01-27T04:20:16.849854+0000","last_undegraded":"2024-01-27T04:20:16.849854+0000","last_fullsized":"2024-01-27T04:20:16.849854+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:07:16.791137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2554,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.862320+0000","last_change":"2024-01-27T03:45:03.762871+0000","last_active":"2024-01-27T04:20:16.862320+0000","last_peered":"2024-01-27T04:20:16.862320+0000","last_clean":"2024-01-27T04:20:16.862320+0000","last_became_active":"2024-01-27T03:45:03.757221+0000","last_became_peered":"2024-01-27T03:45:03.757221+0000","last_unstale":"2024-01-27T04:20:16.862320+0000","last_undegraded":"2024-01-27T04:20:16.862320+0000","last_fullsized":"2024-01-27T04:20:16.862320+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:15:52.156074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2576,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.675809+0000","last_change":"2024-01-27T03:45:01.929664+0000","last_active":"2024-01-27T04:20:16.675809+0000","last_peered":"2024-01-27T04:20:16.675809+0000","last_clean":"2024-01-27T04:20:16.675809+0000","last_became_active":"2024-01-27T03:45:01.916892+0000","last_became_peered":"2024-01-27T03:45:01.916892+0000","last_unstale":"2024-01-27T04:20:16.675809+0000","last_undegraded":"2024-01-27T04:20:16.675809+0000","last_fullsized":"2024-01-27T04:20:16.675809+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:02:49.278494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2570,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.872158+0000","last_change":"2024-01-27T03:45:50.416152+0000","last_active":"2024-01-27T04:20:16.872158+0000","last_peered":"2024-01-27T04:20:16.872158+0000","last_clean":"2024-01-27T04:20:16.872158+0000","last_became_active":"2024-01-27T03:45:50.401687+0000","last_became_peered":"2024-01-27T03:45:50.401687+0000","last_unstale":"2024-01-27T04:20:16.872158+0000","last_undegraded":"2024-01-27T04:20:16.872158+0000","last_fullsized":"2024-01-27T04:20:16.872158+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2572,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.862147+0000","last_change":"2024-01-27T03:45:16.267960+0000","last_active":"2024-01-27T04:20:16.862147+0000","last_peered":"2024-01-27T04:20:16.862147+0000","last_clean":"2024-01-27T04:20:16.862147+0000","last_became_active":"2024-01-27T03:45:16.039935+0000","last_became_peered":"2024-01-27T03:45:16.039935+0000","last_unstale":"2024-01-27T04:20:16.862147+0000","last_undegraded":"2024-01-27T04:20:16.862147+0000","last_fullsized":"2024-01-27T04:20:16.862147+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"48'7","reported_seq":2592,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.664147+0000","last_change":"2024-01-27T03:45:00.305112+0000","last_active":"2024-01-27T04:20:16.664147+0000","last_peered":"2024-01-27T04:20:16.664147+0000","last_clean":"2024-01-27T04:20:16.664147+0000","last_became_active":"2024-01-27T03:45:00.291987+0000","last_became_peered":"2024-01-27T03:45:00.291987+0000","last_unstale":"2024-01-27T04:20:16.664147+0000","last_undegraded":"2024-01-27T04:20:16.664147+0000","last_fullsized":"2024-01-27T04:20:16.664147+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:46:37.876016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2556,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.836922+0000","last_change":"2024-01-27T03:44:59.103125+0000","last_active":"2024-01-27T04:20:16.836922+0000","last_peered":"2024-01-27T04:20:16.836922+0000","last_clean":"2024-01-27T04:20:16.836922+0000","last_became_active":"2024-01-27T03:44:59.080371+0000","last_became_peered":"2024-01-27T03:44:59.080371+0000","last_unstale":"2024-01-27T04:20:16.836922+0000","last_undegraded":"2024-01-27T04:20:16.836922+0000","last_fullsized":"2024-01-27T04:20:16.836922+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:26:17.291036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2552,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:21.897862+0000","last_change":"2024-01-27T04:20:21.897862+0000","last_active":"2024-01-27T04:20:21.897862+0000","last_peered":"2024-01-27T04:20:21.897862+0000","last_clean":"2024-01-27T04:20:21.897862+0000","last_became_active":"2024-01-27T03:45:05.758355+0000","last_became_peered":"2024-01-27T03:45:05.758355+0000","last_unstale":"2024-01-27T04:20:21.897862+0000","last_undegraded":"2024-01-27T04:20:21.897862+0000","last_fullsized":"2024-01-27T04:20:21.897862+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:21.895501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:21.895501+0000","last_clean_scrub_stamp":"2024-01-27T04:20:21.895501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:51:32.538131+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2557,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:22.243451+0000","last_change":"2024-01-27T03:45:11.894875+0000","last_active":"2024-01-27T04:20:22.243451+0000","last_peered":"2024-01-27T04:20:22.243451+0000","last_clean":"2024-01-27T04:20:22.243451+0000","last_became_active":"2024-01-27T03:45:11.884947+0000","last_became_peered":"2024-01-27T03:45:11.884947+0000","last_unstale":"2024-01-27T04:20:22.243451+0000","last_undegraded":"2024-01-27T04:20:22.243451+0000","last_fullsized":"2024-01-27T04:20:22.243451+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'76","reported_seq":2586,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.852478+0000","last_change":"2024-01-27T03:43:25.035922+0000","last_active":"2024-01-27T04:20:16.852478+0000","last_peered":"2024-01-27T04:20:16.852478+0000","last_clean":"2024-01-27T04:20:16.852478+0000","last_became_active":"2024-01-27T03:43:24.404196+0000","last_became_peered":"2024-01-27T03:43:24.404196+0000","last_unstale":"2024-01-27T04:20:16.852478+0000","last_undegraded":"2024-01-27T04:20:16.852478+0000","last_fullsized":"2024-01-27T04:20:16.852478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:13.902947+0000","last_clean_scrub_stamp":"2024-01-27T03:43:13.902947+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2584,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.869864+0000","last_change":"2024-01-27T03:43:30.313647+0000","last_active":"2024-01-27T04:20:16.869864+0000","last_peered":"2024-01-27T04:20:16.869864+0000","last_clean":"2024-01-27T04:20:16.869864+0000","last_became_active":"2024-01-27T03:43:30.303196+0000","last_became_peered":"2024-01-27T03:43:30.303196+0000","last_unstale":"2024-01-27T04:20:16.869864+0000","last_undegraded":"2024-01-27T04:20:16.869864+0000","last_fullsized":"2024-01-27T04:20:16.869864+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2555,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.839672+0000","last_change":"2024-01-27T03:45:06.207200+0000","last_active":"2024-01-27T04:20:16.839672+0000","last_peered":"2024-01-27T04:20:16.839672+0000","last_clean":"2024-01-27T04:20:16.839672+0000","last_became_active":"2024-01-27T03:45:06.197607+0000","last_became_peered":"2024-01-27T03:45:06.197607+0000","last_unstale":"2024-01-27T04:20:16.839672+0000","last_undegraded":"2024-01-27T04:20:16.839672+0000","last_fullsized":"2024-01-27T04:20:16.839672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:30:59.795065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2594,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:26.003610+0000","last_change":"2024-01-27T04:20:26.004361+0000","last_active":"2024-01-27T04:20:26.003610+0000","last_peered":"2024-01-27T04:20:26.003610+0000","last_clean":"2024-01-27T04:20:26.003610+0000","last_became_active":"2024-01-27T03:43:30.033663+0000","last_became_peered":"2024-01-27T03:43:30.033663+0000","last_unstale":"2024-01-27T04:20:26.003610+0000","last_undegraded":"2024-01-27T04:20:26.003610+0000","last_fullsized":"2024-01-27T04:20:26.003610+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:26.002177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:26.002177+0000","last_clean_scrub_stamp":"2024-01-27T04:20:26.002177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:43:06.776948+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2554,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.853095+0000","last_change":"2024-01-27T03:45:07.001823+0000","last_active":"2024-01-27T04:20:16.853095+0000","last_peered":"2024-01-27T04:20:16.853095+0000","last_clean":"2024-01-27T04:20:16.853095+0000","last_became_active":"2024-01-27T03:45:06.986316+0000","last_became_peered":"2024-01-27T03:45:06.986316+0000","last_unstale":"2024-01-27T04:20:16.853095+0000","last_undegraded":"2024-01-27T04:20:16.853095+0000","last_fullsized":"2024-01-27T04:20:16.853095+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:40:42.117379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2588,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.666161+0000","last_change":"2024-01-27T03:43:30.100170+0000","last_active":"2024-01-27T04:20:16.666161+0000","last_peered":"2024-01-27T04:20:16.666161+0000","last_clean":"2024-01-27T04:20:16.666161+0000","last_became_active":"2024-01-27T03:43:30.072010+0000","last_became_peered":"2024-01-27T03:43:30.072010+0000","last_unstale":"2024-01-27T04:20:16.666161+0000","last_undegraded":"2024-01-27T04:20:16.666161+0000","last_fullsized":"2024-01-27T04:20:16.666161+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:17:46.003541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2562,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.859024+0000","last_change":"2024-01-27T03:45:28.248666+0000","last_active":"2024-01-27T04:20:16.859024+0000","last_peered":"2024-01-27T04:20:16.859024+0000","last_clean":"2024-01-27T04:20:16.859024+0000","last_became_active":"2024-01-27T03:45:28.239949+0000","last_became_peered":"2024-01-27T03:45:28.239949+0000","last_unstale":"2024-01-27T04:20:16.859024+0000","last_undegraded":"2024-01-27T04:20:16.859024+0000","last_fullsized":"2024-01-27T04:20:16.859024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2585,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.864735+0000","last_change":"2024-01-27T03:43:30.335032+0000","last_active":"2024-01-27T04:20:16.864735+0000","last_peered":"2024-01-27T04:20:16.864735+0000","last_clean":"2024-01-27T04:20:16.864735+0000","last_became_active":"2024-01-27T03:43:30.329967+0000","last_became_peered":"2024-01-27T03:43:30.329967+0000","last_unstale":"2024-01-27T04:20:16.864735+0000","last_undegraded":"2024-01-27T04:20:16.864735+0000","last_fullsized":"2024-01-27T04:20:16.864735+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2555,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.845632+0000","last_change":"2024-01-27T03:45:03.469717+0000","last_active":"2024-01-27T04:20:16.845632+0000","last_peered":"2024-01-27T04:20:16.845632+0000","last_clean":"2024-01-27T04:20:16.845632+0000","last_became_active":"2024-01-27T03:45:03.435938+0000","last_became_peered":"2024-01-27T03:45:03.435938+0000","last_unstale":"2024-01-27T04:20:16.845632+0000","last_undegraded":"2024-01-27T04:20:16.845632+0000","last_fullsized":"2024-01-27T04:20:16.845632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:31:47.895364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2565,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.859513+0000","last_change":"2024-01-27T03:43:30.340472+0000","last_active":"2024-01-27T04:20:16.859513+0000","last_peered":"2024-01-27T04:20:16.859513+0000","last_clean":"2024-01-27T04:20:16.859513+0000","last_became_active":"2024-01-27T03:43:30.329560+0000","last_became_peered":"2024-01-27T03:43:30.329560+0000","last_unstale":"2024-01-27T04:20:16.859513+0000","last_undegraded":"2024-01-27T04:20:16.859513+0000","last_fullsized":"2024-01-27T04:20:16.859513+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:10:58.217001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2569,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.870411+0000","last_change":"2024-01-27T03:45:11.682176+0000","last_active":"2024-01-27T04:20:16.870411+0000","last_peered":"2024-01-27T04:20:16.870411+0000","last_clean":"2024-01-27T04:20:16.870411+0000","last_became_active":"2024-01-27T03:45:11.344546+0000","last_became_peered":"2024-01-27T03:45:11.344546+0000","last_unstale":"2024-01-27T04:20:16.870411+0000","last_undegraded":"2024-01-27T04:20:16.870411+0000","last_fullsized":"2024-01-27T04:20:16.870411+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2592,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:25.008058+0000","last_change":"2024-01-27T04:20:25.008833+0000","last_active":"2024-01-27T04:20:25.008058+0000","last_peered":"2024-01-27T04:20:25.008058+0000","last_clean":"2024-01-27T04:20:25.008058+0000","last_became_active":"2024-01-27T03:43:30.057792+0000","last_became_peered":"2024-01-27T03:43:30.057792+0000","last_unstale":"2024-01-27T04:20:25.008058+0000","last_undegraded":"2024-01-27T04:20:25.008058+0000","last_fullsized":"2024-01-27T04:20:25.008058+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:25.006592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:25.006592+0000","last_clean_scrub_stamp":"2024-01-27T04:20:25.006592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:02.577153+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2571,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.860347+0000","last_change":"2024-01-27T03:45:12.756405+0000","last_active":"2024-01-27T04:20:16.860347+0000","last_peered":"2024-01-27T04:20:16.860347+0000","last_clean":"2024-01-27T04:20:16.860347+0000","last_became_active":"2024-01-27T03:45:12.751200+0000","last_became_peered":"2024-01-27T03:45:12.751200+0000","last_unstale":"2024-01-27T04:20:16.860347+0000","last_undegraded":"2024-01-27T04:20:16.860347+0000","last_fullsized":"2024-01-27T04:20:16.860347+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2583,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:25.537505+0000","last_change":"2024-01-27T04:20:25.538236+0000","last_active":"2024-01-27T04:20:25.537505+0000","last_peered":"2024-01-27T04:20:25.537505+0000","last_clean":"2024-01-27T04:20:25.537505+0000","last_became_active":"2024-01-27T03:43:30.335129+0000","last_became_peered":"2024-01-27T03:43:30.335129+0000","last_unstale":"2024-01-27T04:20:25.537505+0000","last_undegraded":"2024-01-27T04:20:25.537505+0000","last_fullsized":"2024-01-27T04:20:25.537505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:25.536113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:25.536113+0000","last_clean_scrub_stamp":"2024-01-27T04:20:25.536113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:39:45.332404+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2573,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.876824+0000","last_change":"2024-01-27T03:45:11.743478+0000","last_active":"2024-01-27T04:20:16.876824+0000","last_peered":"2024-01-27T04:20:16.876824+0000","last_clean":"2024-01-27T04:20:16.876824+0000","last_became_active":"2024-01-27T03:45:11.734547+0000","last_became_peered":"2024-01-27T03:45:11.734547+0000","last_unstale":"2024-01-27T04:20:16.876824+0000","last_undegraded":"2024-01-27T04:20:16.876824+0000","last_fullsized":"2024-01-27T04:20:16.876824+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2583,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:24.272619+0000","last_change":"2024-01-27T03:43:30.350497+0000","last_active":"2024-01-27T04:20:24.272619+0000","last_peered":"2024-01-27T04:20:24.272619+0000","last_clean":"2024-01-27T04:20:24.272619+0000","last_became_active":"2024-01-27T03:43:30.345375+0000","last_became_peered":"2024-01-27T03:43:30.345375+0000","last_unstale":"2024-01-27T04:20:24.272619+0000","last_undegraded":"2024-01-27T04:20:24.272619+0000","last_fullsized":"2024-01-27T04:20:24.272619+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:43:28.424223+0000","last_clean_scrub_stamp":"2024-01-27T03:43:28.424223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2574,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.674004+0000","last_change":"2024-01-27T03:45:08.095965+0000","last_active":"2024-01-27T04:20:16.674004+0000","last_peered":"2024-01-27T04:20:16.674004+0000","last_clean":"2024-01-27T04:20:16.674004+0000","last_became_active":"2024-01-27T03:45:08.074261+0000","last_became_peered":"2024-01-27T03:45:08.074261+0000","last_unstale":"2024-01-27T04:20:16.674004+0000","last_undegraded":"2024-01-27T04:20:16.674004+0000","last_fullsized":"2024-01-27T04:20:16.674004+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T15:20:03.504139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2561,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.733985+0000","last_change":"2024-01-27T03:45:29.625526+0000","last_active":"2024-01-27T04:20:16.733985+0000","last_peered":"2024-01-27T04:20:16.733985+0000","last_clean":"2024-01-27T04:20:16.733985+0000","last_became_active":"2024-01-27T03:45:29.620303+0000","last_became_peered":"2024-01-27T03:45:29.620303+0000","last_unstale":"2024-01-27T04:20:16.733985+0000","last_undegraded":"2024-01-27T04:20:16.733985+0000","last_fullsized":"2024-01-27T04:20:16.733985+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2556,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:22.243836+0000","last_change":"2024-01-27T03:45:11.908907+0000","last_active":"2024-01-27T04:20:22.243836+0000","last_peered":"2024-01-27T04:20:22.243836+0000","last_clean":"2024-01-27T04:20:22.243836+0000","last_became_active":"2024-01-27T03:45:11.902875+0000","last_became_peered":"2024-01-27T03:45:11.902875+0000","last_unstale":"2024-01-27T04:20:22.243836+0000","last_undegraded":"2024-01-27T04:20:22.243836+0000","last_fullsized":"2024-01-27T04:20:22.243836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2553,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.857772+0000","last_change":"2024-01-27T03:45:41.083249+0000","last_active":"2024-01-27T04:20:16.857772+0000","last_peered":"2024-01-27T04:20:16.857772+0000","last_clean":"2024-01-27T04:20:16.857772+0000","last_became_active":"2024-01-27T03:45:41.077954+0000","last_became_peered":"2024-01-27T03:45:41.077954+0000","last_unstale":"2024-01-27T04:20:16.857772+0000","last_undegraded":"2024-01-27T04:20:16.857772+0000","last_fullsized":"2024-01-27T04:20:16.857772+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:53:19.777797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2556,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.731803+0000","last_change":"2024-01-27T03:45:13.508628+0000","last_active":"2024-01-27T04:20:16.731803+0000","last_peered":"2024-01-27T04:20:16.731803+0000","last_clean":"2024-01-27T04:20:16.731803+0000","last_became_active":"2024-01-27T03:45:13.503336+0000","last_became_peered":"2024-01-27T03:45:13.503336+0000","last_unstale":"2024-01-27T04:20:16.731803+0000","last_undegraded":"2024-01-27T04:20:16.731803+0000","last_fullsized":"2024-01-27T04:20:16.731803+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2560,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.856959+0000","last_change":"2024-01-27T03:45:06.421706+0000","last_active":"2024-01-27T04:20:16.856959+0000","last_peered":"2024-01-27T04:20:16.856959+0000","last_clean":"2024-01-27T04:20:16.856959+0000","last_became_active":"2024-01-27T03:45:06.412514+0000","last_became_peered":"2024-01-27T03:45:06.412514+0000","last_unstale":"2024-01-27T04:20:16.856959+0000","last_undegraded":"2024-01-27T04:20:16.856959+0000","last_fullsized":"2024-01-27T04:20:16.856959+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2571,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.855575+0000","last_change":"2024-01-27T03:45:05.911725+0000","last_active":"2024-01-27T04:20:16.855575+0000","last_peered":"2024-01-27T04:20:16.855575+0000","last_clean":"2024-01-27T04:20:16.855575+0000","last_became_active":"2024-01-27T03:45:05.905013+0000","last_became_peered":"2024-01-27T03:45:05.905013+0000","last_unstale":"2024-01-27T04:20:16.855575+0000","last_undegraded":"2024-01-27T04:20:16.855575+0000","last_fullsized":"2024-01-27T04:20:16.855575+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2561,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.849906+0000","last_change":"2024-01-27T03:45:09.267062+0000","last_active":"2024-01-27T04:20:16.849906+0000","last_peered":"2024-01-27T04:20:16.849906+0000","last_clean":"2024-01-27T04:20:16.849906+0000","last_became_active":"2024-01-27T03:45:08.513799+0000","last_became_peered":"2024-01-27T03:45:08.513799+0000","last_unstale":"2024-01-27T04:20:16.849906+0000","last_undegraded":"2024-01-27T04:20:16.849906+0000","last_fullsized":"2024-01-27T04:20:16.849906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2574,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.859044+0000","last_change":"2024-01-27T03:45:12.810545+0000","last_active":"2024-01-27T04:20:16.859044+0000","last_peered":"2024-01-27T04:20:16.859044+0000","last_clean":"2024-01-27T04:20:16.859044+0000","last_became_active":"2024-01-27T03:45:12.797603+0000","last_became_peered":"2024-01-27T03:45:12.797603+0000","last_unstale":"2024-01-27T04:20:16.859044+0000","last_undegraded":"2024-01-27T04:20:16.859044+0000","last_fullsized":"2024-01-27T04:20:16.859044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2545,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:22.846209+0000","last_change":"2024-01-27T04:20:22.846967+0000","last_active":"2024-01-27T04:20:22.846209+0000","last_peered":"2024-01-27T04:20:22.846209+0000","last_clean":"2024-01-27T04:20:22.846209+0000","last_became_active":"2024-01-27T03:45:08.146505+0000","last_became_peered":"2024-01-27T03:45:08.146505+0000","last_unstale":"2024-01-27T04:20:22.846209+0000","last_undegraded":"2024-01-27T04:20:22.846209+0000","last_fullsized":"2024-01-27T04:20:22.846209+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:22.844729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:22.844729+0000","last_clean_scrub_stamp":"2024-01-27T04:20:22.844729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:36.926718+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2568,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.868120+0000","last_change":"2024-01-27T03:45:10.396512+0000","last_active":"2024-01-27T04:20:16.868120+0000","last_peered":"2024-01-27T04:20:16.868120+0000","last_clean":"2024-01-27T04:20:16.868120+0000","last_became_active":"2024-01-27T03:45:10.387717+0000","last_became_peered":"2024-01-27T03:45:10.387717+0000","last_unstale":"2024-01-27T04:20:16.868120+0000","last_undegraded":"2024-01-27T04:20:16.868120+0000","last_fullsized":"2024-01-27T04:20:16.868120+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2554,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.855996+0000","last_change":"2024-01-27T03:45:41.124551+0000","last_active":"2024-01-27T04:20:16.855996+0000","last_peered":"2024-01-27T04:20:16.855996+0000","last_clean":"2024-01-27T04:20:16.855996+0000","last_became_active":"2024-01-27T03:45:41.119306+0000","last_became_peered":"2024-01-27T03:45:41.119306+0000","last_unstale":"2024-01-27T04:20:16.855996+0000","last_undegraded":"2024-01-27T04:20:16.855996+0000","last_fullsized":"2024-01-27T04:20:16.855996+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:52:56.858068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2572,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:16.875031+0000","last_change":"2024-01-27T03:45:15.265604+0000","last_active":"2024-01-27T04:20:16.875031+0000","last_peered":"2024-01-27T04:20:16.875031+0000","last_clean":"2024-01-27T04:20:16.875031+0000","last_became_active":"2024-01-27T03:45:15.259193+0000","last_became_peered":"2024-01-27T03:45:15.259193+0000","last_unstale":"2024-01-27T04:20:16.875031+0000","last_undegraded":"2024-01-27T04:20:16.875031+0000","last_fullsized":"2024-01-27T04:20:16.875031+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T03:44:51.643161+0000","last_clean_scrub_stamp":"2024-01-27T03:44:51.643161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313664,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68416,"kb_used_data":5300,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303424,"statfs":{"total":96636764160,"available":96566706176,"internally_reserved":0,"allocated":5427200,"data_stored":2700609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":11.641999999999999,"5min":22.148,"15min":18.57},"min":{"1min":4.3710000000000004,"5min":4.3710000000000004,"15min":4.3710000000000004},"max":{"1min":35.859000000000002,"5min":232.63900000000001,"15min":232.63900000000001},"last":8.7579999999999991},{"interface":"front","average":{"1min":15.961,"5min":23.016999999999999,"15min":19.32},"min":{"1min":4.8449999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"max":{"1min":48.558999999999997,"5min":265.815,"15min":265.815},"last":8.0749999999999993}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":14.598000000000001,"5min":27.135000000000002,"15min":23.359999999999999},"min":{"1min":6.415,"5min":6.415,"15min":5.4749999999999996},"max":{"1min":55.116,"5min":152.09700000000001,"15min":421.90300000000002},"last":22.135999999999999},{"interface":"front","average":{"1min":12.83,"5min":26.446999999999999,"15min":21.591000000000001},"min":{"1min":6.3529999999999998,"5min":5.968,"15min":5.7039999999999997},"max":{"1min":40.463999999999999,"5min":165.53999999999999,"15min":319.13200000000001},"last":11.99}]},{"osd":3,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":16.913,"5min":26.349,"15min":24.771000000000001},"min":{"1min":7.9109999999999996,"5min":7.2750000000000004,"15min":6.0670000000000002},"max":{"1min":54.639000000000003,"5min":176.215,"15min":434.72800000000001},"last":14.067},{"interface":"front","average":{"1min":23.969000000000001,"5min":34.192999999999998,"15min":31.582999999999998},"min":{"1min":10.372999999999999,"5min":10.372999999999999,"15min":8.3019999999999996},"max":{"1min":95.001000000000005,"5min":216.80099999999999,"15min":447.06700000000001},"last":23.117000000000001}]},{"osd":4,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":19.324000000000002,"5min":24.849,"15min":23.100999999999999},"min":{"1min":8.2040000000000006,"5min":7.4240000000000004,"15min":7.0979999999999999},"max":{"1min":59.201000000000001,"5min":152.88,"15min":329.11500000000001},"last":10.816000000000001},{"interface":"front","average":{"1min":18.565999999999999,"5min":24.952999999999999,"15min":21.696999999999999},"min":{"1min":8.4870000000000001,"5min":7.1239999999999997,"15min":5.7839999999999998},"max":{"1min":61.326999999999998,"5min":233.845,"15min":233.845},"last":9.8710000000000004}]},{"osd":5,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":25.951000000000001,"5min":30.747,"15min":26.004000000000001},"min":{"1min":8.7680000000000007,"5min":8.7680000000000007,"15min":7.165},"max":{"1min":101.246,"5min":252.73500000000001,"15min":324.52300000000002},"last":16.501000000000001},{"interface":"front","average":{"1min":27.847999999999999,"5min":38.902999999999999,"15min":30.933},"min":{"1min":13.746,"5min":11.076000000000001,"15min":9.0210000000000008},"max":{"1min":82.462999999999994,"5min":382,"15min":382},"last":20.965}]},{"osd":6,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":17.239999999999998,"5min":29.972999999999999,"15min":28.385000000000002},"min":{"1min":7.335,"5min":7.335,"15min":7.335},"max":{"1min":33.107999999999997,"5min":165.976,"15min":400.93900000000002},"last":16.907},{"interface":"front","average":{"1min":16.268000000000001,"5min":28.919,"15min":27.251000000000001},"min":{"1min":9.4049999999999994,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":26.937999999999999,"5min":217.77699999999999,"15min":684.36699999999996},"last":12.613}]},{"osd":7,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":20.309999999999999,"5min":33.465000000000003,"15min":31.215},"min":{"1min":9.0519999999999996,"5min":8.1389999999999993,"15min":8.1389999999999993},"max":{"1min":50.585000000000001,"5min":242.08199999999999,"15min":443.21499999999997},"last":20.021999999999998},{"interface":"front","average":{"1min":27.695,"5min":32.735999999999997,"15min":28.863},"min":{"1min":12.496,"5min":9.6590000000000007,"15min":8.3309999999999995},"max":{"1min":133.63200000000001,"5min":145.751,"15min":436.80399999999997},"last":18.266999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313662,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76348,"kb_used_data":4908,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295492,"statfs":{"total":96636764160,"available":96558583808,"internally_reserved":0,"allocated":5025792,"data_stored":2302769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":8.7110000000000003,"5min":25.661999999999999,"15min":18.187999999999999},"min":{"1min":4.2939999999999996,"5min":4.2939999999999996,"15min":4.2939999999999996},"max":{"1min":23.433,"5min":445.62400000000002,"15min":445.62400000000002},"last":5.3689999999999998},{"interface":"front","average":{"1min":12.178000000000001,"5min":34.600999999999999,"15min":22.532},"min":{"1min":7.1420000000000003,"5min":6.4370000000000003,"15min":5.3230000000000004},"max":{"1min":16.050000000000001,"5min":974.43299999999999,"15min":974.43299999999999},"last":10.335000000000001}]},{"osd":1,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.199999999999999,"5min":37.404000000000003,"15min":25.739000000000001},"min":{"1min":7.0250000000000004,"5min":7.0250000000000004,"15min":6.6239999999999997},"max":{"1min":19.93,"5min":975.43499999999995,"15min":975.43499999999995},"last":13.151999999999999},{"interface":"front","average":{"1min":12.119999999999999,"5min":28.672000000000001,"15min":22.556000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":5.5899999999999999},"max":{"1min":25.190000000000001,"5min":481.185,"15min":481.185},"last":6.0869999999999997}]},{"osd":3,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":17.236999999999998,"5min":36.584000000000003,"15min":26.488},"min":{"1min":7.665,"5min":7.2709999999999999,"15min":7.2709999999999999},"max":{"1min":59.902999999999999,"5min":980.83500000000004,"15min":980.83500000000004},"last":13.765000000000001},{"interface":"front","average":{"1min":18.442,"5min":41.698,"15min":29.129999999999999},"min":{"1min":11.558999999999999,"5min":10.58,"15min":9.5459999999999994},"max":{"1min":26.449000000000002,"5min":980.17499999999995,"15min":980.17499999999995},"last":17.5}]},{"osd":4,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":12.173999999999999,"5min":27.407,"15min":22.308},"min":{"1min":6.7809999999999997,"5min":6.2750000000000004,"15min":5.819},"max":{"1min":23.806000000000001,"5min":508.601,"15min":508.601},"last":9.0960000000000001},{"interface":"front","average":{"1min":14.736000000000001,"5min":34.642000000000003,"15min":27.102},"min":{"1min":9.7059999999999995,"5min":8.234,"15min":6.6580000000000004},"max":{"1min":27.920999999999999,"5min":492.29199999999997,"15min":492.29199999999997},"last":8.2059999999999995}]},{"osd":5,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":18.317,"5min":48.280000000000001,"15min":33.753999999999998},"min":{"1min":8.9800000000000004,"5min":8.9800000000000004,"15min":8.9740000000000002},"max":{"1min":26.623999999999999,"5min":976.35900000000004,"15min":976.35900000000004},"last":15.162000000000001},{"interface":"front","average":{"1min":12.997,"5min":29.919,"15min":23.949000000000002},"min":{"1min":9.1140000000000008,"5min":6.9610000000000003,"15min":5.758},"max":{"1min":18.027999999999999,"5min":509.71699999999998,"15min":509.71699999999998},"last":10.635}]},{"osd":6,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":33.555999999999997,"15min":28.885000000000002},"min":{"1min":10.654999999999999,"5min":7.8869999999999996,"15min":6.3390000000000004},"max":{"1min":22.977,"5min":526.51400000000001,"15min":526.51400000000001},"last":11.231999999999999},{"interface":"front","average":{"1min":16.966000000000001,"5min":39.537999999999997,"15min":35.607999999999997},"min":{"1min":12.301,"5min":10.784000000000001,"15min":9.4350000000000005},"max":{"1min":23.710999999999999,"5min":533.20799999999997,"15min":533.20799999999997},"last":12.5}]},{"osd":7,"last update":"Sat Jan 27 04:19:26 2024","interfaces":[{"interface":"back","average":{"1min":25.797999999999998,"5min":49.715000000000003,"15min":36.329000000000001},"min":{"1min":14.427,"5min":11.739000000000001,"15min":9.5139999999999993},"max":{"1min":74.819000000000003,"5min":977.13800000000003,"15min":977.13800000000003},"last":15.766},{"interface":"front","average":{"1min":18.117000000000001,"5min":33.683,"15min":29.026},"min":{"1min":11.587999999999999,"5min":8.5850000000000009,"15min":8.5850000000000009},"max":{"1min":52.927999999999997,"5min":549.30100000000004,"15min":549.30100000000004},"last":14.478999999999999}]}]},{"osd":6,"up_from":20,"seq":85899346370,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72184,"kb_used_data":4908,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299656,"statfs":{"total":96636764160,"available":96562847744,"internally_reserved":0,"allocated":5025792,"data_stored":2302769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":9.9730000000000008,"5min":22.446000000000002,"15min":18.524999999999999},"min":{"1min":5.492,"5min":4.6929999999999996,"15min":4.2350000000000003},"max":{"1min":20.303000000000001,"5min":339.68000000000001,"15min":339.68000000000001},"last":13.683},{"interface":"front","average":{"1min":14.538,"5min":32.518000000000001,"15min":26.594999999999999},"min":{"1min":9.9329999999999998,"5min":5.5049999999999999,"15min":4.9749999999999996},"max":{"1min":27.638999999999999,"5min":341.03800000000001,"15min":341.03800000000001},"last":15.276999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":12.906000000000001,"5min":24.239999999999998,"15min":18.077999999999999},"min":{"1min":6.1100000000000003,"5min":5.1970000000000001,"15min":4.6950000000000003},"max":{"1min":35.095999999999997,"5min":250.96000000000001,"15min":250.96000000000001},"last":9.2129999999999992},{"interface":"front","average":{"1min":12.737,"5min":29.638999999999999,"15min":23.449000000000002},"min":{"1min":6.181,"5min":5.0679999999999996,"15min":4.7670000000000003},"max":{"1min":27.792000000000002,"5min":350.74599999999998,"15min":350.74599999999998},"last":10.726000000000001}]},{"osd":2,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":12.606999999999999,"5min":31.344999999999999,"15min":23.207000000000001},"min":{"1min":6.6669999999999998,"5min":6.6669999999999998,"15min":6.0839999999999996},"max":{"1min":25.248999999999999,"5min":349.19299999999998,"15min":349.19299999999998},"last":17.292999999999999},{"interface":"front","average":{"1min":14.715,"5min":35.548000000000002,"15min":29.663},"min":{"1min":9.3729999999999993,"5min":6.5499999999999998,"15min":6.4630000000000001},"max":{"1min":23.071999999999999,"5min":354.80799999999999,"15min":354.80799999999999},"last":14.574999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:43 2024","interfaces":[{"interface":"back","average":{"1min":16.457000000000001,"5min":32.703000000000003,"15min":24.315000000000001},"min":{"1min":6.9299999999999997,"5min":6.9279999999999999,"15min":6.7469999999999999},"max":{"1min":54.694000000000003,"5min":247.446,"15min":265.50299999999999},"last":15.605},{"interface":"front","average":{"1min":20.056999999999999,"5min":34.040999999999997,"15min":28.632999999999999},"min":{"1min":10.635999999999999,"5min":6.4930000000000003,"15min":6.4930000000000003},"max":{"1min":49.180999999999997,"5min":341.68400000000003,"15min":352.79399999999998},"last":19.114999999999998}]},{"osd":4,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":15.965,"5min":27.364999999999998,"15min":24.411000000000001},"min":{"1min":10.279999999999999,"5min":6.7590000000000003,"15min":6.5519999999999996},"max":{"1min":33.625999999999998,"5min":243.96100000000001,"15min":266.12400000000002},"last":16.327000000000002},{"interface":"front","average":{"1min":14.163,"5min":30.145,"15min":27.082000000000001},"min":{"1min":7.359,"5min":6.6859999999999999,"15min":6.0289999999999999},"max":{"1min":29.823,"5min":348.11500000000001,"15min":348.11500000000001},"last":17.957999999999998}]},{"osd":5,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":17.280999999999999,"5min":30.981999999999999,"15min":26.800000000000001},"min":{"1min":11.436999999999999,"5min":7.827,"15min":7.827},"max":{"1min":28.574000000000002,"5min":295.56,"15min":295.56},"last":16.922999999999998},{"interface":"front","average":{"1min":19.364000000000001,"5min":37.106000000000002,"15min":30.992000000000001},"min":{"1min":11.782,"5min":10.959,"15min":8.3569999999999993},"max":{"1min":47.966999999999999,"5min":349.81,"15min":354.55099999999999},"last":23.111999999999998}]},{"osd":7,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":26.315000000000001,"5min":38.106000000000002,"15min":25.777000000000001},"min":{"1min":14.762,"5min":9.7609999999999992,"15min":7.4199999999999999},"max":{"1min":93.060000000000002,"5min":346.697,"15min":346.697},"last":23.741},{"interface":"front","average":{"1min":18.989999999999998,"5min":37.228000000000002,"15min":28.795000000000002},"min":{"1min":11.340999999999999,"5min":7.7670000000000003,"15min":7.7670000000000003},"max":{"1min":49.277000000000001,"5min":352.11900000000003,"15min":369.92899999999997},"last":18.539999999999999}]}]},{"osd":4,"up_from":20,"seq":85899346367,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72184,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299656,"statfs":{"total":96636764160,"available":96562847744,"internally_reserved":0,"allocated":5029888,"data_stored":2302774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":12.260999999999999,"5min":21.777000000000001,"15min":17.795999999999999},"min":{"1min":4.5209999999999999,"5min":4.4020000000000001,"15min":3.782},"max":{"1min":30.248000000000001,"5min":271.60599999999999,"15min":271.60599999999999},"last":14.725},{"interface":"front","average":{"1min":11.218999999999999,"5min":19.111999999999998,"15min":15.164999999999999},"min":{"1min":4.8700000000000001,"5min":4.8700000000000001,"15min":4.7830000000000004},"max":{"1min":36.665999999999997,"5min":141.624,"15min":141.624},"last":8.0220000000000002}]},{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":15.82,"5min":20.286999999999999,"15min":20.946000000000002},"min":{"1min":7.3730000000000002,"5min":5.8159999999999998,"15min":5.8159999999999998},"max":{"1min":44.912999999999997,"5min":228.27600000000001,"15min":228.27600000000001},"last":17.545000000000002},{"interface":"front","average":{"1min":13.35,"5min":23.087,"15min":21.771999999999998},"min":{"1min":7.7060000000000004,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":41.997,"5min":276.93299999999999,"15min":276.93299999999999},"last":11.712999999999999}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":18.422999999999998,"5min":21.227,"15min":22.727},"min":{"1min":7.9669999999999996,"5min":7.8319999999999999,"15min":6.4219999999999997},"max":{"1min":62.054000000000002,"5min":138.13300000000001,"15min":252.38399999999999},"last":16.754999999999999},{"interface":"front","average":{"1min":16.931999999999999,"5min":26.645,"15min":22.036999999999999},"min":{"1min":9.7780000000000005,"5min":7.4930000000000003,"15min":5.8979999999999997},"max":{"1min":48.671999999999997,"5min":305.358,"15min":321.09399999999999},"last":10.837999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":19.376000000000001,"5min":22.815999999999999,"15min":22.629999999999999},"min":{"1min":10.507999999999999,"5min":8.452,"15min":8.1780000000000008},"max":{"1min":61.652999999999999,"5min":89.129999999999995,"15min":195.47800000000001},"last":13.321999999999999},{"interface":"front","average":{"1min":14.627000000000001,"5min":17.146999999999998,"15min":16.039999999999999},"min":{"1min":6.4699999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":91.978999999999999,"5min":120.044,"15min":154.279},"last":9.2029999999999994}]},{"osd":5,"last update":"Sat Jan 27 04:19:38 2024","interfaces":[{"interface":"back","average":{"1min":16.870999999999999,"5min":26.902000000000001,"15min":21.398},"min":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.1779999999999999},"max":{"1min":57.954999999999998,"5min":243.983,"15min":243.983},"last":7.5659999999999998},{"interface":"front","average":{"1min":21.181000000000001,"5min":27.506,"15min":24.664000000000001},"min":{"1min":11.605,"5min":9.7690000000000001,"15min":9.7200000000000006},"max":{"1min":61.289000000000001,"5min":188.453,"15min":191.036},"last":12.281000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":21.027000000000001,"5min":29.692,"15min":27.587},"min":{"1min":8.3409999999999993,"5min":8.3409999999999993,"15min":8.3409999999999993},"max":{"1min":46.170999999999999,"5min":259.351,"15min":259.351},"last":15.954000000000001},{"interface":"front","average":{"1min":24.855,"5min":30.391999999999999,"15min":24.253},"min":{"1min":14.805,"5min":6.9089999999999998,"15min":6.4630000000000001},"max":{"1min":49.533999999999999,"5min":295.00900000000001,"15min":295.00900000000001},"last":19.234999999999999}]},{"osd":7,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":20.744,"5min":25.904,"15min":21.736999999999998},"min":{"1min":10.656000000000001,"5min":8.1850000000000005,"15min":6.4269999999999996},"max":{"1min":52.637999999999998,"5min":206.73400000000001,"15min":206.73400000000001},"last":14.010999999999999},{"interface":"front","average":{"1min":26.753,"5min":32.119,"15min":27.643000000000001},"min":{"1min":15.661,"5min":10.401999999999999,"15min":9.7940000000000005},"max":{"1min":90.658000000000001,"5min":262.28300000000002,"15min":262.28300000000002},"last":18.337}]}]},{"osd":3,"up_from":16,"seq":68719477183,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72180,"kb_used_data":4908,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299660,"statfs":{"total":96636764160,"available":96562851840,"internally_reserved":0,"allocated":5025792,"data_stored":2302769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:36 2024","interfaces":[{"interface":"back","average":{"1min":11.134,"5min":16.122,"15min":16.141999999999999},"min":{"1min":5.3609999999999998,"5min":5.3609999999999998,"15min":4.6310000000000002},"max":{"1min":33.012,"5min":93.953000000000003,"15min":238.642},"last":12.177},{"interface":"front","average":{"1min":14.760999999999999,"5min":20.975000000000001,"15min":20.623999999999999},"min":{"1min":8.0269999999999992,"5min":7.274,"15min":7.274},"max":{"1min":34.912999999999997,"5min":135.18299999999999,"15min":230.04900000000001},"last":14.727}]},{"osd":1,"last update":"Sat Jan 27 04:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.132,"5min":15.403,"15min":15.083},"min":{"1min":5.7380000000000004,"5min":5.0730000000000004,"15min":4.7359999999999998},"max":{"1min":30.003,"5min":103.377,"15min":188.899},"last":7.3360000000000003},{"interface":"front","average":{"1min":11.282999999999999,"5min":16.084,"15min":19.201000000000001},"min":{"1min":5.7759999999999998,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":26.603000000000002,"5min":102.009,"15min":218.97399999999999},"last":10.872999999999999}]},{"osd":2,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":16.824000000000002,"5min":22.969999999999999,"15min":21.879999999999999},"min":{"1min":7.8319999999999999,"5min":7.8319999999999999,"15min":7.8319999999999999},"max":{"1min":44.707000000000001,"5min":130.16499999999999,"15min":322.24599999999998},"last":18.899999999999999},{"interface":"front","average":{"1min":19.797999999999998,"5min":25.126000000000001,"15min":25.841999999999999},"min":{"1min":11,"5min":9.8699999999999992,"15min":9.8699999999999992},"max":{"1min":53.359000000000002,"5min":201.851,"15min":326.04700000000003},"last":19.785}]},{"osd":4,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":22.923999999999999,"5min":27.471,"15min":25.827000000000002},"min":{"1min":11.742000000000001,"5min":10.131,"15min":10.131},"max":{"1min":115.759,"5min":220.51300000000001,"15min":250.255},"last":17.286000000000001},{"interface":"front","average":{"1min":19.719999999999999,"5min":20.579999999999998,"15min":17.524999999999999},"min":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.5049999999999999},"max":{"1min":150.36199999999999,"5min":221.44999999999999,"15min":227.67400000000001},"last":15.554}]},{"osd":5,"last update":"Sat Jan 27 04:19:29 2024","interfaces":[{"interface":"back","average":{"1min":15.736000000000001,"5min":17.591000000000001,"15min":15.472},"min":{"1min":7.2169999999999996,"5min":5.5979999999999999,"15min":4.9359999999999999},"max":{"1min":49.698999999999998,"5min":212.291,"15min":212.291},"last":8.6649999999999991},{"interface":"front","average":{"1min":16.992999999999999,"5min":19.856999999999999,"15min":19.529},"min":{"1min":8.782,"5min":6.742,"15min":6.3550000000000004},"max":{"1min":70.760000000000005,"5min":147.773,"15min":237.16300000000001},"last":9.5519999999999996}]},{"osd":6,"last update":"Sat Jan 27 04:19:57 2024","interfaces":[{"interface":"back","average":{"1min":15.177,"5min":20.619,"15min":19.541},"min":{"1min":8.1270000000000007,"5min":7.4939999999999998,"15min":6.1760000000000002},"max":{"1min":27.283999999999999,"5min":145.55199999999999,"15min":261.56799999999998},"last":11.507999999999999},{"interface":"front","average":{"1min":16.452000000000002,"5min":23.617999999999999,"15min":23.219999999999999},"min":{"1min":11.131,"5min":7.8700000000000001,"15min":7.8700000000000001},"max":{"1min":34.841999999999999,"5min":135.69900000000001,"15min":276.86099999999999},"last":16.154}]},{"osd":7,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":27.474,"5min":29.882999999999999,"15min":26.277999999999999},"min":{"1min":12.359999999999999,"5min":8.0449999999999999,"15min":8.0449999999999999},"max":{"1min":188.46100000000001,"5min":188.46100000000001,"15min":382.57999999999998},"last":18.100999999999999},{"interface":"front","average":{"1min":24.861000000000001,"5min":28.331,"15min":22.138999999999999},"min":{"1min":9.2249999999999996,"5min":9.2249999999999996,"15min":8.9849999999999994},"max":{"1min":222.232,"5min":222.232,"15min":254.12},"last":13.917999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477181,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72180,"kb_used_data":4908,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299660,"statfs":{"total":96636764160,"available":96562851840,"internally_reserved":0,"allocated":5025792,"data_stored":2302769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":11.346,"5min":21.847999999999999,"15min":19.535},"min":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"max":{"1min":28.620000000000001,"5min":197.15600000000001,"15min":286.24799999999999},"last":6.6479999999999997},{"interface":"front","average":{"1min":13.196999999999999,"5min":29.672999999999998,"15min":23.280999999999999},"min":{"1min":6.726,"5min":6.0350000000000001,"15min":4.7960000000000003},"max":{"1min":36.061,"5min":254.524,"15min":267.99299999999999},"last":9.5}]},{"osd":2,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":12.795999999999999,"5min":24.466999999999999,"15min":21.399000000000001},"min":{"1min":6.4829999999999997,"5min":5.984,"15min":5.1449999999999996},"max":{"1min":19.736000000000001,"5min":229.99299999999999,"15min":330.28699999999998},"last":11.186},{"interface":"front","average":{"1min":16.109999999999999,"5min":28.257000000000001,"15min":22.437999999999999},"min":{"1min":10.855,"5min":7.6020000000000003,"15min":6.1669999999999998},"max":{"1min":30.111999999999998,"5min":218.97900000000001,"15min":289.72899999999998},"last":12.789}]},{"osd":3,"last update":"Sat Jan 27 04:19:27 2024","interfaces":[{"interface":"back","average":{"1min":12.762,"5min":23.349,"15min":19.062999999999999},"min":{"1min":6.3230000000000004,"5min":6.3230000000000004,"15min":6.1840000000000002},"max":{"1min":28.111999999999998,"5min":156.90100000000001,"15min":243.779},"last":7.2599999999999998},{"interface":"front","average":{"1min":19.774999999999999,"5min":31.922999999999998,"15min":25.346},"min":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.0860000000000003},"max":{"1min":32.655999999999999,"5min":265.04300000000001,"15min":265.04300000000001},"last":12.185}]},{"osd":4,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":22.652000000000001,"5min":31.033000000000001,"15min":24.866},"min":{"1min":8.2270000000000003,"5min":7.4429999999999996,"15min":7.1870000000000003},"max":{"1min":90.774000000000001,"5min":238.68600000000001,"15min":238.68600000000001},"last":8.9139999999999997},{"interface":"front","average":{"1min":26.318999999999999,"5min":30.364999999999998,"15min":27.841000000000001},"min":{"1min":10.281000000000001,"5min":9.7170000000000005,"15min":9.5559999999999992},"max":{"1min":89.200000000000003,"5min":174.03800000000001,"15min":242.22800000000001},"last":14.939}]},{"osd":5,"last update":"Sat Jan 27 04:19:27 2024","interfaces":[{"interface":"back","average":{"1min":17.667999999999999,"5min":40.302999999999997,"15min":26.126999999999999},"min":{"1min":10.754,"5min":7.0670000000000002,"15min":6.665},"max":{"1min":34.997,"5min":385.50799999999998,"15min":385.50799999999998},"last":10.577},{"interface":"front","average":{"1min":21.75,"5min":42.832999999999998,"15min":28.548999999999999},"min":{"1min":11.871,"5min":9.6639999999999997,"15min":6.5839999999999996},"max":{"1min":47.709000000000003,"5min":337.73700000000002,"15min":337.73700000000002},"last":13.379}]},{"osd":6,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.641999999999999,"5min":39.985999999999997,"15min":31.425000000000001},"min":{"1min":13.096,"5min":10.368,"15min":10.368},"max":{"1min":36.189999999999998,"5min":236.387,"15min":248.78899999999999},"last":15.634},{"interface":"front","average":{"1min":17.960999999999999,"5min":34.942,"15min":28.195},"min":{"1min":7.9359999999999999,"5min":6.4889999999999999,"15min":6.4889999999999999},"max":{"1min":88.625,"5min":252.99700000000001,"15min":381.90800000000002},"last":16.259}]},{"osd":7,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.481000000000002,"5min":32.908000000000001,"15min":28.715},"min":{"1min":8.6880000000000006,"5min":8.6880000000000006,"15min":8.4670000000000005},"max":{"1min":56.813000000000002,"5min":232.21600000000001,"15min":236.81299999999999},"last":14.148999999999999},{"interface":"front","average":{"1min":21.489999999999998,"5min":37.271000000000001,"15min":28.934000000000001},"min":{"1min":12.420999999999999,"5min":10.638,"15min":7.2409999999999997},"max":{"1min":34.862000000000002,"5min":249.29499999999999,"15min":249.29499999999999},"last":16.978000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72572,"kb_used_data":5300,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299268,"statfs":{"total":96636764160,"available":96562450432,"internally_reserved":0,"allocated":5427200,"data_stored":2700609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":21.77,"5min":23.620999999999999,"15min":18.902000000000001},"min":{"1min":5.0140000000000002,"5min":4.4569999999999999,"15min":4.3319999999999999},"max":{"1min":203.28899999999999,"5min":232.74199999999999,"15min":278.56900000000002},"last":11.420999999999999},{"interface":"front","average":{"1min":19.181999999999999,"5min":22.715,"15min":17.931000000000001},"min":{"1min":7.4649999999999999,"5min":5.4530000000000003,"15min":4.8129999999999997},"max":{"1min":115.354,"5min":234.15799999999999,"15min":305.03800000000001},"last":9.6010000000000009}]},{"osd":1,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":31.167000000000002,"5min":26.742000000000001,"15min":23.835999999999999},"min":{"1min":7.3579999999999997,"5min":6.0549999999999997,"15min":6.0549999999999997},"max":{"1min":192.631,"5min":227.602,"15min":304.28300000000002},"last":17.370999999999999},{"interface":"front","average":{"1min":28.222000000000001,"5min":23.238,"15min":20.527000000000001},"min":{"1min":6.5300000000000002,"5min":6.5300000000000002,"15min":5.0199999999999996},"max":{"1min":175.59700000000001,"5min":175.59700000000001,"15min":224.27199999999999},"last":10.225}]},{"osd":2,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":23.504999999999999,"5min":30.835000000000001,"15min":29.007000000000001},"min":{"1min":6.375,"5min":6.375,"15min":6.375},"max":{"1min":105.164,"5min":239.91900000000001,"15min":311.17200000000003},"last":9.5079999999999991},{"interface":"front","average":{"1min":22.550000000000001,"5min":26.795000000000002,"15min":25.391999999999999},"min":{"1min":7.056,"5min":6.8170000000000002,"15min":6.8170000000000002},"max":{"1min":104.66500000000001,"5min":209.00899999999999,"15min":314.78699999999998},"last":8.3510000000000009}]},{"osd":3,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":26.381,"5min":25.306000000000001,"15min":20.504000000000001},"min":{"1min":7.7729999999999997,"5min":6.7690000000000001,"15min":6.274},"max":{"1min":179.31299999999999,"5min":187.23099999999999,"15min":274.71899999999999},"last":8.5190000000000001},{"interface":"front","average":{"1min":24.838000000000001,"5min":28.155999999999999,"15min":24.609999999999999},"min":{"1min":11.1,"5min":8.0670000000000002,"15min":7.3819999999999997},"max":{"1min":108.70099999999999,"5min":175.44800000000001,"15min":233.72800000000001},"last":15.054}]},{"osd":4,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":24.469000000000001,"5min":26.721,"15min":22.181000000000001},"min":{"1min":8.5410000000000004,"5min":6.4269999999999996,"15min":5.9059999999999997},"max":{"1min":132.90299999999999,"5min":185.755,"15min":219.21899999999999},"last":6.75},{"interface":"front","average":{"1min":26.553000000000001,"5min":30.170999999999999,"15min":25.722000000000001},"min":{"1min":9.5739999999999998,"5min":8.7789999999999999,"15min":7.6689999999999996},"max":{"1min":116.852,"5min":174.81,"15min":174.81},"last":18.015000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:32 2024","interfaces":[{"interface":"back","average":{"1min":25.891999999999999,"5min":31.309000000000001,"15min":27.731999999999999},"min":{"1min":10.345000000000001,"5min":6.9299999999999997,"15min":6.9299999999999997},"max":{"1min":96.703999999999994,"5min":234.95099999999999,"15min":259.803},"last":12.959},{"interface":"front","average":{"1min":30.484000000000002,"5min":33.567999999999998,"15min":29.574999999999999},"min":{"1min":10.73,"5min":10.73,"15min":8.2859999999999996},"max":{"1min":98.491,"5min":206.55799999999999,"15min":206.55799999999999},"last":16.48}]},{"osd":7,"last update":"Sat Jan 27 04:20:14 2024","interfaces":[{"interface":"back","average":{"1min":33.548999999999999,"5min":31.448,"15min":26.803000000000001},"min":{"1min":11.276999999999999,"5min":8.0739999999999998,"15min":8.0739999999999998},"max":{"1min":180.07300000000001,"5min":190.39099999999999,"15min":319.75400000000002},"last":20.844999999999999},{"interface":"front","average":{"1min":32.110999999999997,"5min":34.119999999999997,"15min":28.417999999999999},"min":{"1min":10.561999999999999,"5min":9.0800000000000001,"15min":6.6459999999999999},"max":{"1min":217.697,"5min":240.69800000000001,"15min":277.59500000000003},"last":10.833}]}]},{"osd":7,"up_from":13,"seq":55834575293,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72196,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299644,"statfs":{"total":96636764160,"available":96562835456,"internally_reserved":0,"allocated":5029888,"data_stored":2302774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:00 2024","interfaces":[{"interface":"back","average":{"1min":10.483000000000001,"5min":21.335000000000001,"15min":15.542999999999999},"min":{"1min":5.649,"5min":4.7370000000000001,"15min":4.1779999999999999},"max":{"1min":24.736999999999998,"5min":162.80799999999999,"15min":162.80799999999999},"last":8.1370000000000005},{"interface":"front","average":{"1min":13.324,"5min":28.123999999999999,"15min":19.064},"min":{"1min":6.5460000000000003,"5min":4.4020000000000001,"15min":4.4020000000000001},"max":{"1min":22.442,"5min":265.62200000000001,"15min":265.62200000000001},"last":14.942}]},{"osd":1,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":14.460000000000001,"5min":25.378,"15min":21.184000000000001},"min":{"1min":9.5470000000000006,"5min":6.0899999999999999,"15min":6.0899999999999999},"max":{"1min":22.472999999999999,"5min":171.16800000000001,"15min":204.17099999999999},"last":13.555999999999999},{"interface":"front","average":{"1min":8.282,"5min":20.821000000000002,"15min":18.806999999999999},"min":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"max":{"1min":20.916,"5min":192.11000000000001,"15min":323.48599999999999},"last":5.524}]},{"osd":2,"last update":"Sat Jan 27 04:19:28 2024","interfaces":[{"interface":"back","average":{"1min":13.647,"5min":30.683,"15min":21.582999999999998},"min":{"1min":9.0679999999999996,"5min":6.1779999999999999,"15min":6.1269999999999998},"max":{"1min":23.294,"5min":420.65199999999999,"15min":420.65199999999999},"last":9.4350000000000005},{"interface":"front","average":{"1min":13.297000000000001,"5min":28.969000000000001,"15min":22.788},"min":{"1min":6.25,"5min":6.25,"15min":6.25},"max":{"1min":36.289000000000001,"5min":453.39100000000002,"15min":453.39100000000002},"last":10.512}]},{"osd":3,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":14.026,"5min":27.434000000000001,"15min":23.035},"min":{"1min":9.8599999999999994,"5min":8.0830000000000002,"15min":7.6890000000000001},"max":{"1min":23.341999999999999,"5min":226.72300000000001,"15min":226.72300000000001},"last":49.590000000000003},{"interface":"front","average":{"1min":15.734999999999999,"5min":24.466000000000001,"15min":21.619},"min":{"1min":7.6680000000000001,"5min":5.7270000000000003,"15min":5.7270000000000003},"max":{"1min":23.318999999999999,"5min":197.01499999999999,"15min":197.01499999999999},"last":48.740000000000002}]},{"osd":4,"last update":"Sat Jan 27 04:20:11 2024","interfaces":[{"interface":"back","average":{"1min":15.734,"5min":29.295000000000002,"15min":22.515000000000001},"min":{"1min":9.2780000000000005,"5min":8.4749999999999996,"15min":8.3200000000000003},"max":{"1min":48.780000000000001,"5min":227.72300000000001,"15min":227.72300000000001},"last":11.417},{"interface":"front","average":{"1min":18.984000000000002,"5min":32.296999999999997,"15min":24.923999999999999},"min":{"1min":11.102,"5min":9.3059999999999992,"15min":9.3059999999999992},"max":{"1min":55.207000000000001,"5min":230.16,"15min":230.16},"last":15.695}]},{"osd":5,"last update":"Sat Jan 27 04:20:23 2024","interfaces":[{"interface":"back","average":{"1min":20.927,"5min":39.575000000000003,"15min":27.835000000000001},"min":{"1min":10.061,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":113.69799999999999,"5min":276.23500000000001,"15min":276.23500000000001},"last":12.878},{"interface":"front","average":{"1min":23.864000000000001,"5min":36.542000000000002,"15min":29.625},"min":{"1min":12.994999999999999,"5min":10.522,"15min":10.522},"max":{"1min":115.364,"5min":408.774,"15min":408.774},"last":17.529}]},{"osd":6,"last update":"Sat Jan 27 04:19:33 2024","interfaces":[{"interface":"back","average":{"1min":20.045999999999999,"5min":34.045000000000002,"15min":29.82},"min":{"1min":15.138,"5min":10.936,"15min":10.936},"max":{"1min":27.131,"5min":276.92099999999999,"15min":304.952},"last":14.285},{"interface":"front","average":{"1min":14.42,"5min":30.312000000000001,"15min":26.094000000000001},"min":{"1min":7.2199999999999998,"5min":7.2199999999999998,"15min":7.2199999999999998},"max":{"1min":25.401,"5min":302.61700000000002,"15min":302.61700000000002},"last":12.254}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T04:20:28.112 INFO:tasks.ceph:pgid 9.19 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.112 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.112 INFO:tasks.ceph:pgid 9.1b last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.113 INFO:tasks.ceph:pgid 9.1a last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.113 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.113 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.113 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.113 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.113 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.113 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.113 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.113 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-27T03:43:13.902947+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=43, tm_sec=13, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.114 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-27T03:43:28.424223+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=43, tm_sec=28, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.114 INFO:tasks.ceph:pgid 9.8 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.114 INFO:tasks.ceph:pgid 9.b last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.114 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-01-27T03:43:28.424223+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=43, tm_sec=28, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.114 INFO:tasks.ceph:pgid 9.a last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.114 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-27T03:43:28.424223+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=43, tm_sec=28, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.114 INFO:tasks.ceph:pgid 9.9 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.114 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-27T03:43:28.424223+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=43, tm_sec=28, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.114 INFO:tasks.ceph:pgid 9.f last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 9.e last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 9.d last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-27T03:43:28.424223+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=43, tm_sec=28, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 9.1e last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 9.1f last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 9.1c last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 9.1d last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 9.12 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.115 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.116 INFO:tasks.ceph:pgid 9.10 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.116 INFO:tasks.ceph:pgid 9.11 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.116 INFO:tasks.ceph:pgid 9.17 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.116 INFO:tasks.ceph:pgid 9.14 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.116 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2024-01-27T03:44:51.643161+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=3, tm_min=44, tm_sec=51, tm_wday=5, tm_yday=27, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=27, tm_hour=4, tm_min=20, tm_sec=19, tm_wday=5, tm_yday=27, tm_isdst=0) 2024-01-27T04:20:28.116 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-27T04:20:32.988 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:18.944 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:32.993 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:18.949 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:33.020 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:18.976 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:33.038 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:18.983 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:33.040 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:18.986 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:33.045 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:19.001 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:33.048 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:19.005 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:33.058 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:19.003 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:33.067 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:19.023 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:43.015 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:28.970 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:43.016 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:28.972 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:43.031 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:28.988 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:43.062 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:29.007 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:43.063 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:29.019 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:43.064 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:29.020 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:43.066 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:29.011 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:43.077 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:29.033 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:43.079 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:29.024 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:47.480 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:33.436 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:47.529 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:33.485 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:47.533 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:33.479 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:47.537 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:33.482 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:47.547 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:33.503 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:47.551 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:33.497 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:47.595 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:33.539 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:47.595 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:33.543 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:47.600 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:33.557 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:48.117 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-27T04:20:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.223+0000 7f86a25d5640 10 monclient: get_monmap_and_config 2024-01-27T04:20:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.223+0000 7f86a25d5640 10 monclient: build_initial_monmap 2024-01-27T04:20:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.223+0000 7f86a25d5640 10 monclient: monmap: 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:48.223683+0000 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:48.223683+0000 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:48.224 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 1 Processor -- start 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 1 -- start start 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient: init 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f869c0b4b00 0x7f869c0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient(hunting): picked mon.noname-c con 0x7f869c0b4b00 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f869c13f020 0x7f869c13f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient(hunting): picked mon.noname-a con 0x7f869c13f020 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c142060 0x7f869c1424f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient(hunting): picked mon.noname-b con 0x7f869c142060 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c0b44d0 con 0x7f869c13f020 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c0aacc0 con 0x7f869c0b4b00 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0b4ee0 con 0x7f869c142060 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.224+0000 7f86a25d5640 10 monclient(hunting): authenticate will time out at 3731.577938s 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.225+0000 7f869bfff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f869c0b4b00 0x7f869c0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.226+0000 7f869bfff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f869c0b4b00 0x7f869c0b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:37650/0 (socket says 172.21.15.110:37650) 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.226+0000 7f869bfff640 1 -- 172.21.15.110:0/3566400945 learned_addr learned my addr 172.21.15.110:0/3566400945 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.226+0000 7f869bfff640 10 monclient(hunting): get_auth_request con 0x7f869c0b4b00 auth_method 0 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.226+0000 7f869bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.226+0000 7f869bfff640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.226+0000 7f869bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.226+0000 7f869bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.226+0000 7f869bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.226+0000 7f869bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 10 monclient(hunting): handle_auth_done global_id 21001 payload 274 2024-01-27T04:20:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 1 -- 172.21.15.110:0/3566400945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c142060 msgr2=0x7f869c1424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 1 --2- 172.21.15.110:0/3566400945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c142060 0x7f869c1424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 1 -- 172.21.15.110:0/3566400945 >> v2:172.21.15.110:3300/0 conn(0x7f869c13f020 msgr2=0x7f869c13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 1 --2- 172.21.15.110:0/3566400945 >> v2:172.21.15.110:3300/0 conn(0x7f869c13f020 0x7f869c13f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 10 monclient: _finish_hunting 0 2024-01-27T04:20:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 1 monclient: found mon.noname-c 2024-01-27T04:20:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.162:3300/0 2024-01-27T04:20:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.228+0000 7f869bfff640 1 -- 172.21.15.110:0/3566400945 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c13f9d0 con 0x7f869c0b4b00 2024-01-27T04:20:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.230+0000 7f869bfff640 1 --2- 172.21.15.110:0/3566400945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c0b4b00 0x7f869c0b3db0 secure :-1 s=READY pgs=5107 cs=0 l=1 rev1=1 crypto rx=0x7f869c0685c0 tx=0x7f868c02b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c249919fcecc0e62 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 1 -- 172.21.15.110:0/3566400945 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c031740 con 0x7f869c0b4b00 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 10 monclient: dump: 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 10 monclient: _finish_auth 0 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 10 monclient: _check_auth_tickets 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 1 -- 172.21.15.110:0/3566400945 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c0318a0 con 0x7f869c0b4b00 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 5 monclient: authenticate success, global_id 21001 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 1 -- 172.21.15.110:0/3566400945 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c031c30 con 0x7f869c0b4b00 2024-01-27T04:20:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 4 set_mon_vals no callback set 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f869affd640 10 monclient: dump: 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:48.236 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 10 monclient: shutdown 2024-01-27T04:20:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 1 -- 172.21.15.110:0/3566400945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c0b4b00 msgr2=0x7f869c0b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 1 --2- 172.21.15.110:0/3566400945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c0b4b00 0x7f869c0b3db0 secure :-1 s=READY pgs=5107 cs=0 l=1 rev1=1 crypto rx=0x7f869c0685c0 tx=0x7f868c02b7e0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 1 -- 172.21.15.110:0/3566400945 shutdown_connections 2024-01-27T04:20:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 1 --2- 172.21.15.110:0/3566400945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c142060 0x7f869c1424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 1 --2- 172.21.15.110:0/3566400945 >> v2:172.21.15.110:3300/0 conn(0x7f869c13f020 0x7f869c13f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 1 --2- 172.21.15.110:0/3566400945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c0b4b00 0x7f869c0b3db0 unknown :-1 s=CLOSED pgs=5107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.234+0000 7f86a25d5640 1 -- 172.21.15.110:0/3566400945 >> 172.21.15.110:0/3566400945 conn(0x7f869c066490 msgr2=0x7f869c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.235+0000 7f86a25d5640 1 -- 172.21.15.110:0/3566400945 shutdown_connections 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.235+0000 7f86a25d5640 1 -- 172.21.15.110:0/3566400945 wait complete. 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.235+0000 7f86a25d5640 10 monclient: build_initial_monmap 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.235+0000 7f86a25d5640 10 monclient: monmap: 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:48.240 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.235+0000 7f86a25d5640 1 librados: starting msgr at 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.235+0000 7f86a25d5640 1 librados: starting objecter 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 Processor -- start 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 -- start start 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 librados: setting wanted keys 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 librados: calling monclient init 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient: init 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c0b4b00 0x7f869c0b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient(hunting): picked mon.noname-c con 0x7f869c0b4b00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c13f020 0x7f869c1ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient(hunting): picked mon.noname-b con 0x7f869c13f020 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c142060 0x7f869c138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient(hunting): picked mon.noname-a con 0x7f869c142060 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1cebb0 con 0x7f869c142060 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1ce1d0 con 0x7f869c13f020 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1ce4b0 con 0x7f869c0b4b00 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.236+0000 7f86a25d5640 10 monclient(hunting): authenticate will time out at 3731.589794s 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.237+0000 7f869b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c13f020 0x7f869c1ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.237+0000 7f869b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c13f020 0x7f869c1ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:37664/0 (socket says 172.21.15.110:37664) 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.237+0000 7f869b7fe640 1 -- 172.21.15.110:0/2316175791 learned_addr learned my addr 172.21.15.110:0/2316175791 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.237+0000 7f869b7fe640 10 monclient(hunting): get_auth_request con 0x7f869c13f020 auth_method 0 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.237+0000 7f869b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.237+0000 7f869b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.237+0000 7f869b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.238+0000 7f869b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.238+0000 7f869b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.238+0000 7f869b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.239+0000 7f869bfff640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c0b4b00 0x7f869c0b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 10 monclient(hunting): handle_auth_done global_id 21004 payload 824 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 1 -- 172.21.15.110:0/2316175791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c0b4b00 msgr2=0x7f869c0b11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c0b4b00 0x7f869c0b11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 1 -- 172.21.15.110:0/2316175791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c142060 msgr2=0x7f869c138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c142060 0x7f869c138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 10 monclient: _finish_hunting 0 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 1 monclient: found mon.noname-b 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:48.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.240+0000 7f869b7fe640 1 -- 172.21.15.110:0/2316175791 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c0094d0 con 0x7f869c13f020 2024-01-27T04:20:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.242+0000 7f869b7fe640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c13f020 0x7f869c1ccc90 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 crypto rx=0x7f869000b4a0 tx=0x7f869000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.242+0000 7f8698ff9640 10 client.?.objecter ms_handle_connect 0x7f869c13f020 2024-01-27T04:20:48.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.242+0000 7f8698ff9640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 1 -- 172.21.15.110:0/2316175791 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690010610 con 0x7f869c13f020 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 10 monclient: dump: 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:48.247 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 10 monclient: _finish_auth 0 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 10 monclient: _check_auth_tickets 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 1 -- 172.21.15.110:0/2316175791 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8690010770 con 0x7f869c13f020 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f86a25d5640 5 monclient: authenticate success, global_id 21004 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f86a25d5640 10 monclient: _renew_subs 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f86a25d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c1ce790 con 0x7f869c13f020 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 1 -- 172.21.15.110:0/2316175791 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690010b20 con 0x7f869c13f020 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f8698ff9640 10 monclient: dump: 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:48.248 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f86a25d5640 10 client.21004.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f869a7fc640 4 set_mon_vals no callback set 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f86a25d5640 10 monclient: _renew_subs 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f86a25d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c1cc0c0 con 0x7f869c13f020 2024-01-27T04:20:48.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.246+0000 7f86a25d5640 1 librados: init done 2024-01-27T04:20:48.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.247+0000 7f86a25d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:48.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.247+0000 7f86a25d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:48.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.247+0000 7f86a25d5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:48.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.247+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c1cc2e0 con 0x7f869c13f020 2024-01-27T04:20:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.266+0000 7f8698ff9640 1 -- 172.21.15.110:0/2316175791 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7f8690010c80 con 0x7f869c13f020 2024-01-27T04:20:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.267+0000 7f8698ff9640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f868806c9a0 0x7f868806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.267+0000 7f869bfff640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f868806c9a0 0x7f868806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.267+0000 7f869bfff640 10 monclient: get_auth_request con 0x7f868806c9a0 auth_method 0 2024-01-27T04:20:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.267+0000 7f8698ff9640 10 client.21004.objecter ms_handle_connect 0x7f868806c9a0 2024-01-27T04:20:48.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.267+0000 7f869bfff640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f868806c9a0 0x7f868806ee60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f868c035000 tx=0x7f868c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:48.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.495+0000 7f8698ff9640 1 -- 172.21.15.110:0/2316175791 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7f8690058ab0 con 0x7f869c13f020 2024-01-27T04:20:48.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.495+0000 7f8698ff9640 10 client.21004.objecter ms_dispatch 0x7f869c000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:48.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.495+0000 7f8698ff9640 3 client.21004.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:48.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.495+0000 7f8698ff9640 3 client.21004.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:48.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.495+0000 7f8698ff9640 20 client.21004.objecter dump_active .. 0 homeless 2024-01-27T04:20:48.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.505+0000 7f8698ff9640 1 -- 172.21.15.110:0/2316175791 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7f8690054650 con 0x7f869c13f020 2024-01-27T04:20:48.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.505+0000 7f8698ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:48.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.505+0000 7f8698ff9640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:48.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.681+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 --> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f869c0ab250 con 0x7f868806c9a0 2024-01-27T04:20:48.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.687+0000 7f8698ff9640 1 -- 172.21.15.110:0/2316175791 <== mgr.4103 v2:172.21.15.162:6800/3672294946 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151599 (secure 0 0 0) 0x7f868c031be0 con 0x7f868806c9a0 2024-01-27T04:20:48.688 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-01-27T04:20:48.689 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-01-27T04:20:48.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 10 librados: watch_flush enter 2024-01-27T04:20:48.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 10 librados: watch_flush exit 2024-01-27T04:20:48.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 20 client.21004.objecter shutdown clearing up homeless session... 2024-01-27T04:20:48.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 10 client.21004.objecter successfully canceled tick 2024-01-27T04:20:48.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f868806c9a0 msgr2=0x7f868806ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:48.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f868806c9a0 0x7f868806ee60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f868c035000 tx=0x7f868c008030 comp rx=0 tx=0).stop 2024-01-27T04:20:48.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 10 monclient: shutdown 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c13f020 msgr2=0x7f869c1ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c13f020 0x7f869c1ccc90 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 crypto rx=0x7f869000b4a0 tx=0x7f869000b960 comp rx=0 tx=0).stop 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 shutdown_connections 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7f868806c9a0 0x7f868806ee60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f869c142060 0x7f869c138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869c13f020 0x7f869c1ccc90 unknown :-1 s=CLOSED pgs=5108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 --2- 172.21.15.110:0/2316175791 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f869c0b4b00 0x7f869c0b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 >> 172.21.15.110:0/2316175791 conn(0x7f869c066490 msgr2=0x7f869c13cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 shutdown_connections 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.689+0000 7f86a25d5640 1 -- 172.21.15.110:0/2316175791 wait complete. 2024-01-27T04:20:48.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.690+0000 7f86a25d5640 1 librados: shutdown 2024-01-27T04:20:48.701 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":2466,"stamp":"2024-01-27T04:20:47.559381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":578232,"kb_used_data":40056,"kb_used_omap":12,"kb_used_meta":538099,"kb_avail":754396488,"statfs":{"total":773094113280,"available":772502003712,"internally_reserved":0,"allocated":41017344,"data_stored":19217842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":551013936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002808"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2566,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:29.569996+0000","last_change":"2024-01-27T04:20:29.570725+0000","last_active":"2024-01-27T04:20:29.569996+0000","last_peered":"2024-01-27T04:20:29.569996+0000","last_clean":"2024-01-27T04:20:29.569996+0000","last_became_active":"2024-01-27T03:45:12.095646+0000","last_became_peered":"2024-01-27T03:45:12.095646+0000","last_unstale":"2024-01-27T04:20:29.569996+0000","last_undegraded":"2024-01-27T04:20:29.569996+0000","last_fullsized":"2024-01-27T04:20:29.569996+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:29.568578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:29.568578+0000","last_clean_scrub_stamp":"2024-01-27T04:20:29.568578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:58:15.589000+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2578,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:28.789372+0000","last_change":"2024-01-27T04:20:28.790088+0000","last_active":"2024-01-27T04:20:28.789372+0000","last_peered":"2024-01-27T04:20:28.789372+0000","last_clean":"2024-01-27T04:20:28.789372+0000","last_became_active":"2024-01-27T03:45:50.339311+0000","last_became_peered":"2024-01-27T03:45:50.339311+0000","last_unstale":"2024-01-27T04:20:28.789372+0000","last_undegraded":"2024-01-27T04:20:28.789372+0000","last_fullsized":"2024-01-27T04:20:28.789372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:28.787968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:28.787968+0000","last_clean_scrub_stamp":"2024-01-27T04:20:28.787968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:59:34.173551+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2571,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:26.586234+0000","last_change":"2024-01-27T04:20:26.587012+0000","last_active":"2024-01-27T04:20:26.586234+0000","last_peered":"2024-01-27T04:20:26.586234+0000","last_clean":"2024-01-27T04:20:26.586234+0000","last_became_active":"2024-01-27T03:45:01.173317+0000","last_became_peered":"2024-01-27T03:45:01.173317+0000","last_unstale":"2024-01-27T04:20:26.586234+0000","last_undegraded":"2024-01-27T04:20:26.586234+0000","last_fullsized":"2024-01-27T04:20:26.586234+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:26.584771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:26.584771+0000","last_clean_scrub_stamp":"2024-01-27T04:20:26.584771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:23:21.913916+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2562,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:32.545646+0000","last_change":"2024-01-27T04:20:32.546409+0000","last_active":"2024-01-27T04:20:32.545646+0000","last_peered":"2024-01-27T04:20:32.545646+0000","last_clean":"2024-01-27T04:20:32.545646+0000","last_became_active":"2024-01-27T03:45:08.187722+0000","last_became_peered":"2024-01-27T03:45:08.187722+0000","last_unstale":"2024-01-27T04:20:32.545646+0000","last_undegraded":"2024-01-27T04:20:32.545646+0000","last_fullsized":"2024-01-27T04:20:32.545646+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:32.544216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:32.544216+0000","last_clean_scrub_stamp":"2024-01-27T04:20:32.544216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:32:54.657109+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2564,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:29.556444+0000","last_change":"2024-01-27T04:20:29.556444+0000","last_active":"2024-01-27T04:20:29.556444+0000","last_peered":"2024-01-27T04:20:29.556444+0000","last_clean":"2024-01-27T04:20:29.556444+0000","last_became_active":"2024-01-27T03:45:03.757221+0000","last_became_peered":"2024-01-27T03:45:03.757221+0000","last_unstale":"2024-01-27T04:20:29.556444+0000","last_undegraded":"2024-01-27T04:20:29.556444+0000","last_fullsized":"2024-01-27T04:20:29.556444+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:29.554167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:29.554167+0000","last_clean_scrub_stamp":"2024-01-27T04:20:29.554167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:04:50.862547+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2586,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:30.420210+0000","last_change":"2024-01-27T04:20:30.421014+0000","last_active":"2024-01-27T04:20:30.420210+0000","last_peered":"2024-01-27T04:20:30.420210+0000","last_clean":"2024-01-27T04:20:30.420210+0000","last_became_active":"2024-01-27T03:45:01.916892+0000","last_became_peered":"2024-01-27T03:45:01.916892+0000","last_unstale":"2024-01-27T04:20:30.420210+0000","last_undegraded":"2024-01-27T04:20:30.420210+0000","last_fullsized":"2024-01-27T04:20:30.420210+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:30.418801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:30.418801+0000","last_clean_scrub_stamp":"2024-01-27T04:20:30.418801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:21:42.048705+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2578,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:29.795979+0000","last_change":"2024-01-27T04:20:29.796700+0000","last_active":"2024-01-27T04:20:29.795979+0000","last_peered":"2024-01-27T04:20:29.795979+0000","last_clean":"2024-01-27T04:20:29.795979+0000","last_became_active":"2024-01-27T03:45:50.401687+0000","last_became_peered":"2024-01-27T03:45:50.401687+0000","last_unstale":"2024-01-27T04:20:29.795979+0000","last_undegraded":"2024-01-27T04:20:29.795979+0000","last_fullsized":"2024-01-27T04:20:29.795979+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:29.794543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:29.794543+0000","last_clean_scrub_stamp":"2024-01-27T04:20:29.794543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:12:20.317504+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2580,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:27.810322+0000","last_change":"2024-01-27T04:20:27.811056+0000","last_active":"2024-01-27T04:20:27.810322+0000","last_peered":"2024-01-27T04:20:27.810322+0000","last_clean":"2024-01-27T04:20:27.810322+0000","last_became_active":"2024-01-27T03:45:16.039935+0000","last_became_peered":"2024-01-27T03:45:16.039935+0000","last_unstale":"2024-01-27T04:20:27.810322+0000","last_undegraded":"2024-01-27T04:20:27.810322+0000","last_fullsized":"2024-01-27T04:20:27.810322+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:27.808886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:27.808886+0000","last_clean_scrub_stamp":"2024-01-27T04:20:27.808886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:07:09.321582+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"1334'8","reported_seq":2602,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:29.438312+0000","last_change":"2024-01-27T04:20:29.439273+0000","last_active":"2024-01-27T04:20:29.438312+0000","last_peered":"2024-01-27T04:20:29.438312+0000","last_clean":"2024-01-27T04:20:29.438312+0000","last_became_active":"2024-01-27T03:45:00.291987+0000","last_became_peered":"2024-01-27T03:45:00.291987+0000","last_unstale":"2024-01-27T04:20:29.438312+0000","last_undegraded":"2024-01-27T04:20:29.438312+0000","last_fullsized":"2024-01-27T04:20:29.438312+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"1334'8","last_scrub_stamp":"2024-01-27T04:20:29.436900+0000","last_deep_scrub":"1334'8","last_deep_scrub_stamp":"2024-01-27T04:20:29.436900+0000","last_clean_scrub_stamp":"2024-01-27T04:20:29.436900+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:08:00.189923+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2565,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:27.579213+0000","last_change":"2024-01-27T04:20:27.579950+0000","last_active":"2024-01-27T04:20:27.579213+0000","last_peered":"2024-01-27T04:20:27.579213+0000","last_clean":"2024-01-27T04:20:27.579213+0000","last_became_active":"2024-01-27T03:44:59.080371+0000","last_became_peered":"2024-01-27T03:44:59.080371+0000","last_unstale":"2024-01-27T04:20:27.579213+0000","last_undegraded":"2024-01-27T04:20:27.579213+0000","last_fullsized":"2024-01-27T04:20:27.579213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:27.577842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:27.577842+0000","last_clean_scrub_stamp":"2024-01-27T04:20:27.577842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:40:28.592536+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2552,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:21.897862+0000","last_change":"2024-01-27T04:20:21.897862+0000","last_active":"2024-01-27T04:20:21.897862+0000","last_peered":"2024-01-27T04:20:21.897862+0000","last_clean":"2024-01-27T04:20:21.897862+0000","last_became_active":"2024-01-27T03:45:05.758355+0000","last_became_peered":"2024-01-27T03:45:05.758355+0000","last_unstale":"2024-01-27T04:20:21.897862+0000","last_undegraded":"2024-01-27T04:20:21.897862+0000","last_fullsized":"2024-01-27T04:20:21.897862+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:21.895501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:21.895501+0000","last_clean_scrub_stamp":"2024-01-27T04:20:21.895501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:51:32.538131+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2565,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:27.526132+0000","last_change":"2024-01-27T04:20:27.526872+0000","last_active":"2024-01-27T04:20:27.526132+0000","last_peered":"2024-01-27T04:20:27.526132+0000","last_clean":"2024-01-27T04:20:27.526132+0000","last_became_active":"2024-01-27T03:45:11.884947+0000","last_became_peered":"2024-01-27T03:45:11.884947+0000","last_unstale":"2024-01-27T04:20:27.526132+0000","last_undegraded":"2024-01-27T04:20:27.526132+0000","last_fullsized":"2024-01-27T04:20:27.526132+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:27.524636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:27.524636+0000","last_clean_scrub_stamp":"2024-01-27T04:20:27.524636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:17:22.179035+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"1334'78","reported_seq":2596,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:23.387278+0000","last_change":"2024-01-27T04:20:23.388661+0000","last_active":"2024-01-27T04:20:23.387278+0000","last_peered":"2024-01-27T04:20:23.387278+0000","last_clean":"2024-01-27T04:20:23.387278+0000","last_became_active":"2024-01-27T03:43:24.404196+0000","last_became_peered":"2024-01-27T03:43:24.404196+0000","last_unstale":"2024-01-27T04:20:23.387278+0000","last_undegraded":"2024-01-27T04:20:23.387278+0000","last_fullsized":"2024-01-27T04:20:23.387278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"1334'78","last_scrub_stamp":"2024-01-27T04:20:23.385827+0000","last_deep_scrub":"1334'78","last_deep_scrub_stamp":"2024-01-27T04:20:23.385827+0000","last_clean_scrub_stamp":"2024-01-27T04:20:23.385827+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T16:10:04.065347+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2592,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:26.984985+0000","last_change":"2024-01-27T04:20:26.985751+0000","last_active":"2024-01-27T04:20:26.984985+0000","last_peered":"2024-01-27T04:20:26.984985+0000","last_clean":"2024-01-27T04:20:26.984985+0000","last_became_active":"2024-01-27T03:43:30.303196+0000","last_became_peered":"2024-01-27T03:43:30.303196+0000","last_unstale":"2024-01-27T04:20:26.984985+0000","last_undegraded":"2024-01-27T04:20:26.984985+0000","last_fullsized":"2024-01-27T04:20:26.984985+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:26.983506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:26.983506+0000","last_clean_scrub_stamp":"2024-01-27T04:20:26.983506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:33:00.011203+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2564,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:30.557517+0000","last_change":"2024-01-27T04:20:30.558261+0000","last_active":"2024-01-27T04:20:30.557517+0000","last_peered":"2024-01-27T04:20:30.557517+0000","last_clean":"2024-01-27T04:20:30.557517+0000","last_became_active":"2024-01-27T03:45:06.197607+0000","last_became_peered":"2024-01-27T03:45:06.197607+0000","last_unstale":"2024-01-27T04:20:30.557517+0000","last_undegraded":"2024-01-27T04:20:30.557517+0000","last_fullsized":"2024-01-27T04:20:30.557517+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:30.556008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:30.556008+0000","last_clean_scrub_stamp":"2024-01-27T04:20:30.556008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T11:52:11.598261+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2594,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:26.003610+0000","last_change":"2024-01-27T04:20:26.004361+0000","last_active":"2024-01-27T04:20:26.003610+0000","last_peered":"2024-01-27T04:20:26.003610+0000","last_clean":"2024-01-27T04:20:26.003610+0000","last_became_active":"2024-01-27T03:43:30.033663+0000","last_became_peered":"2024-01-27T03:43:30.033663+0000","last_unstale":"2024-01-27T04:20:26.003610+0000","last_undegraded":"2024-01-27T04:20:26.003610+0000","last_fullsized":"2024-01-27T04:20:26.003610+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:26.002177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:26.002177+0000","last_clean_scrub_stamp":"2024-01-27T04:20:26.002177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:43:06.776948+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2563,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:31.591251+0000","last_change":"2024-01-27T04:20:31.591990+0000","last_active":"2024-01-27T04:20:31.591251+0000","last_peered":"2024-01-27T04:20:31.591251+0000","last_clean":"2024-01-27T04:20:31.591251+0000","last_became_active":"2024-01-27T03:45:06.986316+0000","last_became_peered":"2024-01-27T03:45:06.986316+0000","last_unstale":"2024-01-27T04:20:31.591251+0000","last_undegraded":"2024-01-27T04:20:31.591251+0000","last_fullsized":"2024-01-27T04:20:31.591251+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:31.589807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:31.589807+0000","last_clean_scrub_stamp":"2024-01-27T04:20:31.589807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T13:39:33.136457+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2597,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:28.370381+0000","last_change":"2024-01-27T04:20:28.371138+0000","last_active":"2024-01-27T04:20:28.370381+0000","last_peered":"2024-01-27T04:20:28.370381+0000","last_clean":"2024-01-27T04:20:28.370381+0000","last_became_active":"2024-01-27T03:43:30.072010+0000","last_became_peered":"2024-01-27T03:43:30.072010+0000","last_unstale":"2024-01-27T04:20:28.370381+0000","last_undegraded":"2024-01-27T04:20:28.370381+0000","last_fullsized":"2024-01-27T04:20:28.370381+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:28.368915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:28.368915+0000","last_clean_scrub_stamp":"2024-01-27T04:20:28.368915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T08:08:10.410368+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2570,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:26.315776+0000","last_change":"2024-01-27T04:20:26.316545+0000","last_active":"2024-01-27T04:20:26.315776+0000","last_peered":"2024-01-27T04:20:26.315776+0000","last_clean":"2024-01-27T04:20:26.315776+0000","last_became_active":"2024-01-27T03:45:28.239949+0000","last_became_peered":"2024-01-27T03:45:28.239949+0000","last_unstale":"2024-01-27T04:20:26.315776+0000","last_undegraded":"2024-01-27T04:20:26.315776+0000","last_fullsized":"2024-01-27T04:20:26.315776+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:26.314341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:26.314341+0000","last_clean_scrub_stamp":"2024-01-27T04:20:26.314341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:02:25.206748+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2593,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:28.039173+0000","last_change":"2024-01-27T04:20:28.040465+0000","last_active":"2024-01-27T04:20:28.039173+0000","last_peered":"2024-01-27T04:20:28.039173+0000","last_clean":"2024-01-27T04:20:28.039173+0000","last_became_active":"2024-01-27T03:43:30.329967+0000","last_became_peered":"2024-01-27T03:43:30.329967+0000","last_unstale":"2024-01-27T04:20:28.039173+0000","last_undegraded":"2024-01-27T04:20:28.039173+0000","last_fullsized":"2024-01-27T04:20:28.039173+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:28.036872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:28.036872+0000","last_clean_scrub_stamp":"2024-01-27T04:20:28.036872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:36:12.577927+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2564,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:28.564419+0000","last_change":"2024-01-27T04:20:28.565203+0000","last_active":"2024-01-27T04:20:28.564419+0000","last_peered":"2024-01-27T04:20:28.564419+0000","last_clean":"2024-01-27T04:20:28.564419+0000","last_became_active":"2024-01-27T03:45:03.435938+0000","last_became_peered":"2024-01-27T03:45:03.435938+0000","last_unstale":"2024-01-27T04:20:28.564419+0000","last_undegraded":"2024-01-27T04:20:28.564419+0000","last_fullsized":"2024-01-27T04:20:28.564419+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:28.562993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:28.562993+0000","last_clean_scrub_stamp":"2024-01-27T04:20:28.562993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:09:02.290342+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2574,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:26.771625+0000","last_change":"2024-01-27T04:20:26.776104+0000","last_active":"2024-01-27T04:20:26.771625+0000","last_peered":"2024-01-27T04:20:26.771625+0000","last_clean":"2024-01-27T04:20:26.771625+0000","last_became_active":"2024-01-27T03:43:30.329560+0000","last_became_peered":"2024-01-27T03:43:30.329560+0000","last_unstale":"2024-01-27T04:20:26.771625+0000","last_undegraded":"2024-01-27T04:20:26.771625+0000","last_fullsized":"2024-01-27T04:20:26.771625+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:26.767313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:26.767313+0000","last_clean_scrub_stamp":"2024-01-27T04:20:26.767313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T15:48:58.475040+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2577,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:25.873592+0000","last_change":"2024-01-27T04:20:25.874366+0000","last_active":"2024-01-27T04:20:25.873592+0000","last_peered":"2024-01-27T04:20:25.873592+0000","last_clean":"2024-01-27T04:20:25.873592+0000","last_became_active":"2024-01-27T03:45:11.344546+0000","last_became_peered":"2024-01-27T03:45:11.344546+0000","last_unstale":"2024-01-27T04:20:25.873592+0000","last_undegraded":"2024-01-27T04:20:25.873592+0000","last_fullsized":"2024-01-27T04:20:25.873592+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:25.872165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:25.872165+0000","last_clean_scrub_stamp":"2024-01-27T04:20:25.872165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:29:23.036185+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2592,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:25.008058+0000","last_change":"2024-01-27T04:20:25.008833+0000","last_active":"2024-01-27T04:20:25.008058+0000","last_peered":"2024-01-27T04:20:25.008058+0000","last_clean":"2024-01-27T04:20:25.008058+0000","last_became_active":"2024-01-27T03:43:30.057792+0000","last_became_peered":"2024-01-27T03:43:30.057792+0000","last_unstale":"2024-01-27T04:20:25.008058+0000","last_undegraded":"2024-01-27T04:20:25.008058+0000","last_fullsized":"2024-01-27T04:20:25.008058+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:25.006592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:25.006592+0000","last_clean_scrub_stamp":"2024-01-27T04:20:25.006592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:42:02.577153+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2579,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:26.854378+0000","last_change":"2024-01-27T04:20:26.855243+0000","last_active":"2024-01-27T04:20:26.854378+0000","last_peered":"2024-01-27T04:20:26.854378+0000","last_clean":"2024-01-27T04:20:26.854378+0000","last_became_active":"2024-01-27T03:45:12.751200+0000","last_became_peered":"2024-01-27T03:45:12.751200+0000","last_unstale":"2024-01-27T04:20:26.854378+0000","last_undegraded":"2024-01-27T04:20:26.854378+0000","last_fullsized":"2024-01-27T04:20:26.854378+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:26.849625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:26.849625+0000","last_clean_scrub_stamp":"2024-01-27T04:20:26.849625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T05:47:36.670124+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2583,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:25.537505+0000","last_change":"2024-01-27T04:20:25.538236+0000","last_active":"2024-01-27T04:20:25.537505+0000","last_peered":"2024-01-27T04:20:25.537505+0000","last_clean":"2024-01-27T04:20:25.537505+0000","last_became_active":"2024-01-27T03:43:30.335129+0000","last_became_peered":"2024-01-27T03:43:30.335129+0000","last_unstale":"2024-01-27T04:20:25.537505+0000","last_undegraded":"2024-01-27T04:20:25.537505+0000","last_fullsized":"2024-01-27T04:20:25.537505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:25.536113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:25.536113+0000","last_clean_scrub_stamp":"2024-01-27T04:20:25.536113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:39:45.332404+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2581,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:30.125476+0000","last_change":"2024-01-27T04:20:30.126709+0000","last_active":"2024-01-27T04:20:30.125476+0000","last_peered":"2024-01-27T04:20:30.125476+0000","last_clean":"2024-01-27T04:20:30.125476+0000","last_became_active":"2024-01-27T03:45:11.734547+0000","last_became_peered":"2024-01-27T03:45:11.734547+0000","last_unstale":"2024-01-27T04:20:30.125476+0000","last_undegraded":"2024-01-27T04:20:30.125476+0000","last_fullsized":"2024-01-27T04:20:30.125476+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:30.121203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:30.121203+0000","last_clean_scrub_stamp":"2024-01-27T04:20:30.121203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:09:11.361743+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2591,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:29.027285+0000","last_change":"2024-01-27T04:20:29.027998+0000","last_active":"2024-01-27T04:20:29.027285+0000","last_peered":"2024-01-27T04:20:29.027285+0000","last_clean":"2024-01-27T04:20:29.027285+0000","last_became_active":"2024-01-27T03:43:30.345375+0000","last_became_peered":"2024-01-27T03:43:30.345375+0000","last_unstale":"2024-01-27T04:20:29.027285+0000","last_undegraded":"2024-01-27T04:20:29.027285+0000","last_fullsized":"2024-01-27T04:20:29.027285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:29.025871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:29.025871+0000","last_clean_scrub_stamp":"2024-01-27T04:20:29.025871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:44:24.831039+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2583,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:31.444566+0000","last_change":"2024-01-27T04:20:31.445301+0000","last_active":"2024-01-27T04:20:31.444566+0000","last_peered":"2024-01-27T04:20:31.444566+0000","last_clean":"2024-01-27T04:20:31.444566+0000","last_became_active":"2024-01-27T03:45:08.074261+0000","last_became_peered":"2024-01-27T03:45:08.074261+0000","last_unstale":"2024-01-27T04:20:31.444566+0000","last_undegraded":"2024-01-27T04:20:31.444566+0000","last_fullsized":"2024-01-27T04:20:31.444566+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:31.443205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:31.443205+0000","last_clean_scrub_stamp":"2024-01-27T04:20:31.443205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T07:37:43.416068+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2569,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:31.594781+0000","last_change":"2024-01-27T04:20:31.596036+0000","last_active":"2024-01-27T04:20:31.594781+0000","last_peered":"2024-01-27T04:20:31.594781+0000","last_clean":"2024-01-27T04:20:31.594781+0000","last_became_active":"2024-01-27T03:45:29.620303+0000","last_became_peered":"2024-01-27T03:45:29.620303+0000","last_unstale":"2024-01-27T04:20:31.594781+0000","last_undegraded":"2024-01-27T04:20:31.594781+0000","last_fullsized":"2024-01-27T04:20:31.594781+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:31.592535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:31.592535+0000","last_clean_scrub_stamp":"2024-01-27T04:20:31.592535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T10:57:19.305576+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2564,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:28.619309+0000","last_change":"2024-01-27T04:20:28.620041+0000","last_active":"2024-01-27T04:20:28.619309+0000","last_peered":"2024-01-27T04:20:28.619309+0000","last_clean":"2024-01-27T04:20:28.619309+0000","last_became_active":"2024-01-27T03:45:11.902875+0000","last_became_peered":"2024-01-27T03:45:11.902875+0000","last_unstale":"2024-01-27T04:20:28.619309+0000","last_undegraded":"2024-01-27T04:20:28.619309+0000","last_fullsized":"2024-01-27T04:20:28.619309+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:28.617872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:28.617872+0000","last_clean_scrub_stamp":"2024-01-27T04:20:28.617872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:33:23.049944+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2562,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:33.554913+0000","last_change":"2024-01-27T04:20:33.555689+0000","last_active":"2024-01-27T04:20:33.554913+0000","last_peered":"2024-01-27T04:20:33.554913+0000","last_clean":"2024-01-27T04:20:33.554913+0000","last_became_active":"2024-01-27T03:45:41.077954+0000","last_became_peered":"2024-01-27T03:45:41.077954+0000","last_unstale":"2024-01-27T04:20:33.554913+0000","last_undegraded":"2024-01-27T04:20:33.554913+0000","last_fullsized":"2024-01-27T04:20:33.554913+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:33.553437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:33.553437+0000","last_clean_scrub_stamp":"2024-01-27T04:20:33.553437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T14:21:25.048313+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2564,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:30.612530+0000","last_change":"2024-01-27T04:20:30.613299+0000","last_active":"2024-01-27T04:20:30.612530+0000","last_peered":"2024-01-27T04:20:30.612530+0000","last_clean":"2024-01-27T04:20:30.612530+0000","last_became_active":"2024-01-27T03:45:13.503336+0000","last_became_peered":"2024-01-27T03:45:13.503336+0000","last_unstale":"2024-01-27T04:20:30.612530+0000","last_undegraded":"2024-01-27T04:20:30.612530+0000","last_fullsized":"2024-01-27T04:20:30.612530+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:30.611052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:30.611052+0000","last_clean_scrub_stamp":"2024-01-27T04:20:30.611052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T14:30:56.367529+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2568,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:24.326616+0000","last_change":"2024-01-27T04:20:24.327359+0000","last_active":"2024-01-27T04:20:24.326616+0000","last_peered":"2024-01-27T04:20:24.326616+0000","last_clean":"2024-01-27T04:20:24.326616+0000","last_became_active":"2024-01-27T03:45:06.412514+0000","last_became_peered":"2024-01-27T03:45:06.412514+0000","last_unstale":"2024-01-27T04:20:24.326616+0000","last_undegraded":"2024-01-27T04:20:24.326616+0000","last_fullsized":"2024-01-27T04:20:24.326616+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:24.325104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:24.325104+0000","last_clean_scrub_stamp":"2024-01-27T04:20:24.325104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T04:45:06.808494+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2579,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:23.838452+0000","last_change":"2024-01-27T04:20:23.839200+0000","last_active":"2024-01-27T04:20:23.838452+0000","last_peered":"2024-01-27T04:20:23.838452+0000","last_clean":"2024-01-27T04:20:23.838452+0000","last_became_active":"2024-01-27T03:45:05.905013+0000","last_became_peered":"2024-01-27T03:45:05.905013+0000","last_unstale":"2024-01-27T04:20:23.838452+0000","last_undegraded":"2024-01-27T04:20:23.838452+0000","last_fullsized":"2024-01-27T04:20:23.838452+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:23.835697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:23.835697+0000","last_clean_scrub_stamp":"2024-01-27T04:20:23.835697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T12:12:59.358231+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2569,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:25.295428+0000","last_change":"2024-01-27T04:20:25.296183+0000","last_active":"2024-01-27T04:20:25.295428+0000","last_peered":"2024-01-27T04:20:25.295428+0000","last_clean":"2024-01-27T04:20:25.295428+0000","last_became_active":"2024-01-27T03:45:08.513799+0000","last_became_peered":"2024-01-27T03:45:08.513799+0000","last_unstale":"2024-01-27T04:20:25.295428+0000","last_undegraded":"2024-01-27T04:20:25.295428+0000","last_fullsized":"2024-01-27T04:20:25.295428+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:25.293936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:25.293936+0000","last_clean_scrub_stamp":"2024-01-27T04:20:25.293936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:41:30.443926+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2582,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:31.134202+0000","last_change":"2024-01-27T04:20:31.134988+0000","last_active":"2024-01-27T04:20:31.134202+0000","last_peered":"2024-01-27T04:20:31.134202+0000","last_clean":"2024-01-27T04:20:31.134202+0000","last_became_active":"2024-01-27T03:45:12.797603+0000","last_became_peered":"2024-01-27T03:45:12.797603+0000","last_unstale":"2024-01-27T04:20:31.134202+0000","last_undegraded":"2024-01-27T04:20:31.134202+0000","last_fullsized":"2024-01-27T04:20:31.134202+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:31.129373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:31.129373+0000","last_clean_scrub_stamp":"2024-01-27T04:20:31.129373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T09:46:49.537048+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2545,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:22.846209+0000","last_change":"2024-01-27T04:20:22.846967+0000","last_active":"2024-01-27T04:20:22.846209+0000","last_peered":"2024-01-27T04:20:22.846209+0000","last_clean":"2024-01-27T04:20:22.846209+0000","last_became_active":"2024-01-27T03:45:08.146505+0000","last_became_peered":"2024-01-27T03:45:08.146505+0000","last_unstale":"2024-01-27T04:20:22.846209+0000","last_undegraded":"2024-01-27T04:20:22.846209+0000","last_fullsized":"2024-01-27T04:20:22.846209+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:22.844729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:22.844729+0000","last_clean_scrub_stamp":"2024-01-27T04:20:22.844729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T06:22:36.926718+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2576,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:24.892721+0000","last_change":"2024-01-27T04:20:24.893498+0000","last_active":"2024-01-27T04:20:24.892721+0000","last_peered":"2024-01-27T04:20:24.892721+0000","last_clean":"2024-01-27T04:20:24.892721+0000","last_became_active":"2024-01-27T03:45:10.387717+0000","last_became_peered":"2024-01-27T03:45:10.387717+0000","last_unstale":"2024-01-27T04:20:24.892721+0000","last_undegraded":"2024-01-27T04:20:24.892721+0000","last_fullsized":"2024-01-27T04:20:24.892721+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:24.889340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:24.889340+0000","last_clean_scrub_stamp":"2024-01-27T04:20:24.889340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T14:37:13.389889+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2563,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:34.599611+0000","last_change":"2024-01-27T04:20:34.600346+0000","last_active":"2024-01-27T04:20:34.599611+0000","last_peered":"2024-01-27T04:20:34.599611+0000","last_clean":"2024-01-27T04:20:34.599611+0000","last_became_active":"2024-01-27T03:45:41.119306+0000","last_became_peered":"2024-01-27T03:45:41.119306+0000","last_unstale":"2024-01-27T04:20:34.599611+0000","last_undegraded":"2024-01-27T04:20:34.599611+0000","last_fullsized":"2024-01-27T04:20:34.599611+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:34.598212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:34.598212+0000","last_clean_scrub_stamp":"2024-01-27T04:20:34.598212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T15:25:32.287316+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2580,"reported_epoch":1334,"state":"active+clean","last_fresh":"2024-01-27T04:20:32.107611+0000","last_change":"2024-01-27T04:20:32.108368+0000","last_active":"2024-01-27T04:20:32.107611+0000","last_peered":"2024-01-27T04:20:32.107611+0000","last_clean":"2024-01-27T04:20:32.107611+0000","last_became_active":"2024-01-27T03:45:15.259193+0000","last_became_peered":"2024-01-27T03:45:15.259193+0000","last_unstale":"2024-01-27T04:20:32.107611+0000","last_undegraded":"2024-01-27T04:20:32.107611+0000","last_fullsized":"2024-01-27T04:20:32.107611+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-27T04:20:32.106172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-27T04:20:32.106172+0000","last_clean_scrub_stamp":"2024-01-27T04:20:32.106172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-28T15:48:33.068360+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68412,"kb_used_data":5300,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303428,"statfs":{"total":96636764160,"available":96566710272,"internally_reserved":0,"allocated":5427200,"data_stored":2700609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":11.641999999999999,"5min":22.148,"15min":18.57},"min":{"1min":4.3710000000000004,"5min":4.3710000000000004,"15min":4.3710000000000004},"max":{"1min":35.859000000000002,"5min":232.63900000000001,"15min":232.63900000000001},"last":14.092000000000001},{"interface":"front","average":{"1min":15.961,"5min":23.016999999999999,"15min":19.32},"min":{"1min":4.8449999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"max":{"1min":48.558999999999997,"5min":265.815,"15min":265.815},"last":9.0540000000000003}]},{"osd":2,"last update":"Sat Jan 27 04:20:29 2024","interfaces":[{"interface":"back","average":{"1min":16.123999999999999,"5min":24.398,"15min":23.172000000000001},"min":{"1min":6.4459999999999997,"5min":6.415,"15min":5.4749999999999996},"max":{"1min":26.824999999999999,"5min":112.26300000000001,"15min":421.90300000000002},"last":11.032999999999999},{"interface":"front","average":{"1min":11.49,"5min":22.777000000000001,"15min":21.501000000000001},"min":{"1min":6.056,"5min":6.056,"15min":5.7759999999999998},"max":{"1min":22.477,"5min":165.53999999999999,"15min":319.13200000000001},"last":7.2949999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:20:29 2024","interfaces":[{"interface":"back","average":{"1min":15.544,"5min":23.673999999999999,"15min":23.587},"min":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":6.0670000000000002},"max":{"1min":37.106000000000002,"5min":176.215,"15min":434.72800000000001},"last":12.087999999999999},{"interface":"front","average":{"1min":23.459,"5min":31.998999999999999,"15min":29.588000000000001},"min":{"1min":13.013999999999999,"5min":10.372999999999999,"15min":8.3019999999999996},"max":{"1min":58.091999999999999,"5min":216.80099999999999,"15min":447.06700000000001},"last":12.706}]},{"osd":4,"last update":"Sat Jan 27 04:20:32 2024","interfaces":[{"interface":"back","average":{"1min":22.029,"5min":22.952999999999999,"15min":23.155999999999999},"min":{"1min":6.9459999999999997,"5min":6.9459999999999997,"15min":6.9459999999999997},"max":{"1min":69.856999999999999,"5min":136.81700000000001,"15min":329.11500000000001},"last":13.375},{"interface":"front","average":{"1min":18.408999999999999,"5min":22.452000000000002,"15min":21.492000000000001},"min":{"1min":7.7889999999999997,"5min":7.1239999999999997,"15min":5.7839999999999998},"max":{"1min":70.998000000000005,"5min":233.845,"15min":233.845},"last":8.0969999999999995}]},{"osd":5,"last update":"Sat Jan 27 04:20:29 2024","interfaces":[{"interface":"back","average":{"1min":23.006,"5min":28.719000000000001,"15min":26.145},"min":{"1min":12.018000000000001,"5min":8.7680000000000007,"15min":7.165},"max":{"1min":53.686,"5min":252.73500000000001,"15min":324.52300000000002},"last":16.323},{"interface":"front","average":{"1min":26.84,"5min":34.109999999999999,"15min":30.361999999999998},"min":{"1min":12.288,"5min":11.403,"15min":9.0210000000000008},"max":{"1min":59.972999999999999,"5min":382,"15min":382},"last":18.085999999999999}]},{"osd":6,"last update":"Sat Jan 27 04:20:29 2024","interfaces":[{"interface":"back","average":{"1min":19.359999999999999,"5min":27.332999999999998,"15min":28.183},"min":{"1min":11.247999999999999,"5min":7.335,"15min":7.335},"max":{"1min":39.523000000000003,"5min":165.976,"15min":400.93900000000002},"last":16.972999999999999},{"interface":"front","average":{"1min":16.277999999999999,"5min":26.361000000000001,"15min":27.074000000000002},"min":{"1min":8.6029999999999998,"5min":7.2220000000000004,"15min":7.2220000000000004},"max":{"1min":41.863,"5min":217.77699999999999,"15min":684.36699999999996},"last":9.7240000000000002}]},{"osd":7,"last update":"Sat Jan 27 04:20:29 2024","interfaces":[{"interface":"back","average":{"1min":22.507999999999999,"5min":31.106000000000002,"15min":30.004999999999999},"min":{"1min":8.9909999999999997,"5min":8.1389999999999993,"15min":8.1389999999999993},"max":{"1min":52.353000000000002,"5min":242.08199999999999,"15min":443.21499999999997},"last":14.747999999999999},{"interface":"front","average":{"1min":28.042999999999999,"5min":31.969000000000001,"15min":28.893999999999998},"min":{"1min":10.039999999999999,"5min":10.039999999999999,"15min":8.3309999999999995},"max":{"1min":62.857999999999997,"5min":145.751,"15min":436.80399999999997},"last":18.890999999999998}]}]},{"osd":2,"up_from":21,"seq":90194313666,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76340,"kb_used_data":4908,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295500,"statfs":{"total":96636764160,"available":96558592000,"internally_reserved":0,"allocated":5025792,"data_stored":2302769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":8.7110000000000003,"5min":25.661999999999999,"15min":18.187999999999999},"min":{"1min":4.2939999999999996,"5min":4.2939999999999996,"15min":4.2939999999999996},"max":{"1min":23.433,"5min":445.62400000000002,"15min":445.62400000000002},"last":5.4610000000000003},{"interface":"front","average":{"1min":12.178000000000001,"5min":34.600999999999999,"15min":22.532},"min":{"1min":7.1420000000000003,"5min":6.4370000000000003,"15min":5.3230000000000004},"max":{"1min":16.050000000000001,"5min":974.43299999999999,"15min":974.43299999999999},"last":10.505000000000001}]},{"osd":1,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.199999999999999,"5min":37.404000000000003,"15min":25.739000000000001},"min":{"1min":7.0250000000000004,"5min":7.0250000000000004,"15min":6.6239999999999997},"max":{"1min":19.93,"5min":975.43499999999995,"15min":975.43499999999995},"last":12.866},{"interface":"front","average":{"1min":12.119999999999999,"5min":28.672000000000001,"15min":22.556000000000001},"min":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":5.5899999999999999},"max":{"1min":25.190000000000001,"5min":481.185,"15min":481.185},"last":8.5370000000000008}]},{"osd":3,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":17.236999999999998,"5min":36.584000000000003,"15min":26.488},"min":{"1min":7.665,"5min":7.2709999999999999,"15min":7.2709999999999999},"max":{"1min":59.902999999999999,"5min":980.83500000000004,"15min":980.83500000000004},"last":15.252000000000001},{"interface":"front","average":{"1min":18.442,"5min":41.698,"15min":29.129999999999999},"min":{"1min":11.558999999999999,"5min":10.58,"15min":9.5459999999999994},"max":{"1min":26.449000000000002,"5min":980.17499999999995,"15min":980.17499999999995},"last":18.545000000000002}]},{"osd":4,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":12.173999999999999,"5min":27.407,"15min":22.308},"min":{"1min":6.7809999999999997,"5min":6.2750000000000004,"15min":5.819},"max":{"1min":23.806000000000001,"5min":508.601,"15min":508.601},"last":9.7089999999999996},{"interface":"front","average":{"1min":14.736000000000001,"5min":34.642000000000003,"15min":27.102},"min":{"1min":9.7059999999999995,"5min":8.234,"15min":6.6580000000000004},"max":{"1min":27.920999999999999,"5min":492.29199999999997,"15min":492.29199999999997},"last":12.147}]},{"osd":5,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":18.317,"5min":48.280000000000001,"15min":33.753999999999998},"min":{"1min":8.9800000000000004,"5min":8.9800000000000004,"15min":8.9740000000000002},"max":{"1min":26.623999999999999,"5min":976.35900000000004,"15min":976.35900000000004},"last":19.306000000000001},{"interface":"front","average":{"1min":12.997,"5min":29.919,"15min":23.949000000000002},"min":{"1min":9.1140000000000008,"5min":6.9610000000000003,"15min":5.758},"max":{"1min":18.027999999999999,"5min":509.71699999999998,"15min":509.71699999999998},"last":11.531000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:53 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":33.555999999999997,"15min":28.885000000000002},"min":{"1min":10.654999999999999,"5min":7.8869999999999996,"15min":6.3390000000000004},"max":{"1min":22.977,"5min":526.51400000000001,"15min":526.51400000000001},"last":14.539},{"interface":"front","average":{"1min":16.966000000000001,"5min":39.537999999999997,"15min":35.607999999999997},"min":{"1min":12.301,"5min":10.784000000000001,"15min":9.4350000000000005},"max":{"1min":23.710999999999999,"5min":533.20799999999997,"15min":533.20799999999997},"last":17.664999999999999}]},{"osd":7,"last update":"Sat Jan 27 04:20:27 2024","interfaces":[{"interface":"back","average":{"1min":19.602,"5min":49.439,"15min":35.194000000000003},"min":{"1min":11.771000000000001,"5min":11.739000000000001,"15min":9.5139999999999993},"max":{"1min":29.960000000000001,"5min":977.13800000000003,"15min":977.13800000000003},"last":16.538},{"interface":"front","average":{"1min":17.631,"5min":33.307000000000002,"15min":27.628},"min":{"1min":10.959,"5min":8.5850000000000009,"15min":8.5850000000000009},"max":{"1min":28.212,"5min":549.30100000000004,"15min":549.30100000000004},"last":15.936}]}]},{"osd":6,"up_from":20,"seq":85899346374,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72180,"kb_used_data":4908,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299660,"statfs":{"total":96636764160,"available":96562851840,"internally_reserved":0,"allocated":5025792,"data_stored":2302769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":9.9730000000000008,"5min":22.446000000000002,"15min":18.524999999999999},"min":{"1min":5.492,"5min":4.6929999999999996,"15min":4.2350000000000003},"max":{"1min":20.303000000000001,"5min":339.68000000000001,"15min":339.68000000000001},"last":5.8129999999999997},{"interface":"front","average":{"1min":14.538,"5min":32.518000000000001,"15min":26.594999999999999},"min":{"1min":9.9329999999999998,"5min":5.5049999999999999,"15min":4.9749999999999996},"max":{"1min":27.638999999999999,"5min":341.03800000000001,"15min":341.03800000000001},"last":12.545999999999999}]},{"osd":1,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":12.906000000000001,"5min":24.239999999999998,"15min":18.077999999999999},"min":{"1min":6.1100000000000003,"5min":5.1970000000000001,"15min":4.6950000000000003},"max":{"1min":35.095999999999997,"5min":250.96000000000001,"15min":250.96000000000001},"last":9.0830000000000002},{"interface":"front","average":{"1min":12.737,"5min":29.638999999999999,"15min":23.449000000000002},"min":{"1min":6.181,"5min":5.0679999999999996,"15min":4.7670000000000003},"max":{"1min":27.792000000000002,"5min":350.74599999999998,"15min":350.74599999999998},"last":10.962999999999999}]},{"osd":2,"last update":"Sat Jan 27 04:20:10 2024","interfaces":[{"interface":"back","average":{"1min":12.606999999999999,"5min":31.344999999999999,"15min":23.207000000000001},"min":{"1min":6.6669999999999998,"5min":6.6669999999999998,"15min":6.0839999999999996},"max":{"1min":25.248999999999999,"5min":349.19299999999998,"15min":349.19299999999998},"last":8.1969999999999992},{"interface":"front","average":{"1min":14.715,"5min":35.548000000000002,"15min":29.663},"min":{"1min":9.3729999999999993,"5min":6.5499999999999998,"15min":6.4630000000000001},"max":{"1min":23.071999999999999,"5min":354.80799999999999,"15min":354.80799999999999},"last":16.574999999999999}]},{"osd":3,"last update":"Sat Jan 27 04:19:43 2024","interfaces":[{"interface":"back","average":{"1min":16.457000000000001,"5min":32.703000000000003,"15min":24.315000000000001},"min":{"1min":6.9299999999999997,"5min":6.9279999999999999,"15min":6.7469999999999999},"max":{"1min":54.694000000000003,"5min":247.446,"15min":265.50299999999999},"last":9.8480000000000008},{"interface":"front","average":{"1min":20.056999999999999,"5min":34.040999999999997,"15min":28.632999999999999},"min":{"1min":10.635999999999999,"5min":6.4930000000000003,"15min":6.4930000000000003},"max":{"1min":49.180999999999997,"5min":341.68400000000003,"15min":352.79399999999998},"last":14.363}]},{"osd":4,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":15.965,"5min":27.364999999999998,"15min":24.411000000000001},"min":{"1min":10.279999999999999,"5min":6.7590000000000003,"15min":6.5519999999999996},"max":{"1min":33.625999999999998,"5min":243.96100000000001,"15min":266.12400000000002},"last":11.847},{"interface":"front","average":{"1min":14.163,"5min":30.145,"15min":27.082000000000001},"min":{"1min":7.359,"5min":6.6859999999999999,"15min":6.0289999999999999},"max":{"1min":29.823,"5min":348.11500000000001,"15min":348.11500000000001},"last":15.847}]},{"osd":5,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":17.280999999999999,"5min":30.981999999999999,"15min":26.800000000000001},"min":{"1min":11.436999999999999,"5min":7.827,"15min":7.827},"max":{"1min":28.574000000000002,"5min":295.56,"15min":295.56},"last":13.709},{"interface":"front","average":{"1min":19.364000000000001,"5min":37.106000000000002,"15min":30.992000000000001},"min":{"1min":11.782,"5min":10.959,"15min":8.3569999999999993},"max":{"1min":47.966999999999999,"5min":349.81,"15min":354.55099999999999},"last":17.82}]},{"osd":7,"last update":"Sat Jan 27 04:20:01 2024","interfaces":[{"interface":"back","average":{"1min":26.315000000000001,"5min":38.106000000000002,"15min":25.777000000000001},"min":{"1min":14.762,"5min":9.7609999999999992,"15min":7.4199999999999999},"max":{"1min":93.060000000000002,"5min":346.697,"15min":346.697},"last":18.771999999999998},{"interface":"front","average":{"1min":18.989999999999998,"5min":37.228000000000002,"15min":28.795000000000002},"min":{"1min":11.340999999999999,"5min":7.7670000000000003,"15min":7.7670000000000003},"max":{"1min":49.277000000000001,"5min":352.11900000000003,"15min":369.92899999999997},"last":13.164}]}]},{"osd":4,"up_from":20,"seq":85899346371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72184,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299656,"statfs":{"total":96636764160,"available":96562847744,"internally_reserved":0,"allocated":5029888,"data_stored":2302774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":12.260999999999999,"5min":21.777000000000001,"15min":17.795999999999999},"min":{"1min":4.5209999999999999,"5min":4.4020000000000001,"15min":3.782},"max":{"1min":30.248000000000001,"5min":271.60599999999999,"15min":271.60599999999999},"last":12.063000000000001},{"interface":"front","average":{"1min":11.218999999999999,"5min":19.111999999999998,"15min":15.164999999999999},"min":{"1min":4.8700000000000001,"5min":4.8700000000000001,"15min":4.7830000000000004},"max":{"1min":36.665999999999997,"5min":141.624,"15min":141.624},"last":12.725}]},{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":15.82,"5min":20.286999999999999,"15min":20.946000000000002},"min":{"1min":7.3730000000000002,"5min":5.8159999999999998,"15min":5.8159999999999998},"max":{"1min":44.912999999999997,"5min":228.27600000000001,"15min":228.27600000000001},"last":13.494999999999999},{"interface":"front","average":{"1min":13.35,"5min":23.087,"15min":21.771999999999998},"min":{"1min":7.7060000000000004,"5min":5.7290000000000001,"15min":5.7290000000000001},"max":{"1min":41.997,"5min":276.93299999999999,"15min":276.93299999999999},"last":14.643000000000001}]},{"osd":2,"last update":"Sat Jan 27 04:20:32 2024","interfaces":[{"interface":"back","average":{"1min":18.852,"5min":20.943000000000001,"15min":22.337},"min":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":6.4219999999999997},"max":{"1min":45.558999999999997,"5min":138.13300000000001,"15min":252.38399999999999},"last":15.332000000000001},{"interface":"front","average":{"1min":16.148,"5min":24.170999999999999,"15min":21.956},"min":{"1min":6.9050000000000002,"5min":6.9050000000000002,"15min":6.2160000000000002},"max":{"1min":50.484000000000002,"5min":305.358,"15min":321.09399999999999},"last":15.962}]},{"osd":3,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":19.376000000000001,"5min":22.815999999999999,"15min":22.629999999999999},"min":{"1min":10.507999999999999,"5min":8.452,"15min":8.1780000000000008},"max":{"1min":61.652999999999999,"5min":89.129999999999995,"15min":195.47800000000001},"last":18.408000000000001},{"interface":"front","average":{"1min":14.627000000000001,"5min":17.146999999999998,"15min":16.039999999999999},"min":{"1min":6.4699999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"max":{"1min":91.978999999999999,"5min":120.044,"15min":154.279},"last":7.7969999999999997}]},{"osd":5,"last update":"Sat Jan 27 04:20:38 2024","interfaces":[{"interface":"back","average":{"1min":13.705,"5min":21.489000000000001,"15min":20.870000000000001},"min":{"1min":6.4649999999999999,"5min":6.4649999999999999,"15min":6.1779999999999999},"max":{"1min":60.003999999999998,"5min":114.408,"15min":243.983},"last":9.2119999999999997},{"interface":"front","average":{"1min":19.858000000000001,"5min":24.631,"15min":24.262},"min":{"1min":11.571999999999999,"5min":9.8740000000000006,"15min":9.7690000000000001},"max":{"1min":59.655000000000001,"5min":188.453,"15min":191.036},"last":10.695}]},{"osd":6,"last update":"Sat Jan 27 04:20:15 2024","interfaces":[{"interface":"back","average":{"1min":21.027000000000001,"5min":29.692,"15min":27.587},"min":{"1min":8.3409999999999993,"5min":8.3409999999999993,"15min":8.3409999999999993},"max":{"1min":46.170999999999999,"5min":259.351,"15min":259.351},"last":17.477},{"interface":"front","average":{"1min":24.855,"5min":30.391999999999999,"15min":24.253},"min":{"1min":14.805,"5min":6.9089999999999998,"15min":6.4630000000000001},"max":{"1min":49.533999999999999,"5min":295.00900000000001,"15min":295.00900000000001},"last":19.286999999999999}]},{"osd":7,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":20.744,"5min":25.904,"15min":21.736999999999998},"min":{"1min":10.656000000000001,"5min":8.1850000000000005,"15min":6.4269999999999996},"max":{"1min":52.637999999999998,"5min":206.73400000000001,"15min":206.73400000000001},"last":10.032},{"interface":"front","average":{"1min":26.753,"5min":32.119,"15min":27.643000000000001},"min":{"1min":15.661,"5min":10.401999999999999,"15min":9.7940000000000005},"max":{"1min":90.658000000000001,"5min":262.28300000000002,"15min":262.28300000000002},"last":20.158000000000001}]}]},{"osd":3,"up_from":16,"seq":68719477187,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72180,"kb_used_data":4908,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299660,"statfs":{"total":96636764160,"available":96562851840,"internally_reserved":0,"allocated":5025792,"data_stored":2302769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:41 2024","interfaces":[{"interface":"back","average":{"1min":12.031000000000001,"5min":14.529999999999999,"15min":14.499000000000001},"min":{"1min":5.2220000000000004,"5min":5.2220000000000004,"15min":4.6310000000000002},"max":{"1min":31.291,"5min":67.843000000000004,"15min":188.81800000000001},"last":10.625},{"interface":"front","average":{"1min":15.131,"5min":19.148,"15min":18.695},"min":{"1min":7.7720000000000002,"5min":7.274,"15min":7.274},"max":{"1min":29.184999999999999,"5min":99.090999999999994,"15min":198.392},"last":11.371}]},{"osd":1,"last update":"Sat Jan 27 04:20:20 2024","interfaces":[{"interface":"back","average":{"1min":10.132,"5min":15.403,"15min":15.083},"min":{"1min":5.7380000000000004,"5min":5.0730000000000004,"15min":4.7359999999999998},"max":{"1min":30.003,"5min":103.377,"15min":188.899},"last":7.6269999999999998},{"interface":"front","average":{"1min":11.282999999999999,"5min":16.084,"15min":19.201000000000001},"min":{"1min":5.7759999999999998,"5min":5.5709999999999997,"15min":5.5709999999999997},"max":{"1min":26.603000000000002,"5min":102.009,"15min":218.97399999999999},"last":14.725}]},{"osd":2,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":16.824000000000002,"5min":22.969999999999999,"15min":21.879999999999999},"min":{"1min":7.8319999999999999,"5min":7.8319999999999999,"15min":7.8319999999999999},"max":{"1min":44.707000000000001,"5min":130.16499999999999,"15min":322.24599999999998},"last":17.093},{"interface":"front","average":{"1min":19.797999999999998,"5min":25.126000000000001,"15min":25.841999999999999},"min":{"1min":11,"5min":9.8699999999999992,"15min":9.8699999999999992},"max":{"1min":53.359000000000002,"5min":201.851,"15min":326.04700000000003},"last":18.431000000000001}]},{"osd":4,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":22.923999999999999,"5min":27.471,"15min":25.827000000000002},"min":{"1min":11.742000000000001,"5min":10.131,"15min":10.131},"max":{"1min":115.759,"5min":220.51300000000001,"15min":250.255},"last":21.587},{"interface":"front","average":{"1min":19.719999999999999,"5min":20.579999999999998,"15min":17.524999999999999},"min":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.5049999999999999},"max":{"1min":150.36199999999999,"5min":221.44999999999999,"15min":227.67400000000001},"last":9.2409999999999997}]},{"osd":5,"last update":"Sat Jan 27 04:20:33 2024","interfaces":[{"interface":"back","average":{"1min":11.763,"5min":17.606999999999999,"15min":14.901999999999999},"min":{"1min":6.5149999999999997,"5min":5.7050000000000001,"15min":4.9359999999999999},"max":{"1min":21.134,"5min":212.291,"15min":212.291},"last":8.6310000000000002},{"interface":"front","average":{"1min":15.974,"5min":20.024999999999999,"15min":18.498999999999999},"min":{"1min":9.5519999999999996,"5min":8.782,"15min":6.3550000000000004},"max":{"1min":25.773,"5min":147.773,"15min":202.096},"last":12.242000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:57 2024","interfaces":[{"interface":"back","average":{"1min":15.177,"5min":20.619,"15min":19.541},"min":{"1min":8.1270000000000007,"5min":7.4939999999999998,"15min":6.1760000000000002},"max":{"1min":27.283999999999999,"5min":145.55199999999999,"15min":261.56799999999998},"last":14.053000000000001},{"interface":"front","average":{"1min":16.452000000000002,"5min":23.617999999999999,"15min":23.219999999999999},"min":{"1min":11.131,"5min":7.8700000000000001,"15min":7.8700000000000001},"max":{"1min":34.841999999999999,"5min":135.69900000000001,"15min":276.86099999999999},"last":16.039999999999999}]},{"osd":7,"last update":"Sat Jan 27 04:19:45 2024","interfaces":[{"interface":"back","average":{"1min":27.474,"5min":29.882999999999999,"15min":26.277999999999999},"min":{"1min":12.359999999999999,"5min":8.0449999999999999,"15min":8.0449999999999999},"max":{"1min":188.46100000000001,"5min":188.46100000000001,"15min":382.57999999999998},"last":22.443999999999999},{"interface":"front","average":{"1min":24.861000000000001,"5min":28.331,"15min":22.138999999999999},"min":{"1min":9.2249999999999996,"5min":9.2249999999999996,"15min":8.9849999999999994},"max":{"1min":222.232,"5min":222.232,"15min":254.12},"last":9.968}]}]},{"osd":0,"up_from":16,"seq":68719477185,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72180,"kb_used_data":4908,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299660,"statfs":{"total":96636764160,"available":96562851840,"internally_reserved":0,"allocated":5025792,"data_stored":2302769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":11.346,"5min":21.847999999999999,"15min":19.535},"min":{"1min":4.3319999999999999,"5min":4.3319999999999999,"15min":4.3319999999999999},"max":{"1min":28.620000000000001,"5min":197.15600000000001,"15min":286.24799999999999},"last":6.319},{"interface":"front","average":{"1min":13.196999999999999,"5min":29.672999999999998,"15min":23.280999999999999},"min":{"1min":6.726,"5min":6.0350000000000001,"15min":4.7960000000000003},"max":{"1min":36.061,"5min":254.524,"15min":267.99299999999999},"last":8.2940000000000005}]},{"osd":2,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":12.795999999999999,"5min":24.466999999999999,"15min":21.399000000000001},"min":{"1min":6.4829999999999997,"5min":5.984,"15min":5.1449999999999996},"max":{"1min":19.736000000000001,"5min":229.99299999999999,"15min":330.28699999999998},"last":11.766999999999999},{"interface":"front","average":{"1min":16.109999999999999,"5min":28.257000000000001,"15min":22.437999999999999},"min":{"1min":10.855,"5min":7.6020000000000003,"15min":6.1669999999999998},"max":{"1min":30.111999999999998,"5min":218.97900000000001,"15min":289.72899999999998},"last":9.577}]},{"osd":3,"last update":"Sat Jan 27 04:20:28 2024","interfaces":[{"interface":"back","average":{"1min":10.957000000000001,"5min":21.960000000000001,"15min":17.786000000000001},"min":{"1min":7.2599999999999998,"5min":6.3230000000000004,"15min":6.1840000000000002},"max":{"1min":22.344000000000001,"5min":144.62799999999999,"15min":243.779},"last":7.6260000000000003},{"interface":"front","average":{"1min":17.309000000000001,"5min":30.963999999999999,"15min":24.407},"min":{"1min":10.608000000000001,"5min":8.3689999999999998,"15min":8.0860000000000003},"max":{"1min":54.368000000000002,"5min":265.04300000000001,"15min":265.04300000000001},"last":15.443}]},{"osd":4,"last update":"Sat Jan 27 04:19:46 2024","interfaces":[{"interface":"back","average":{"1min":22.652000000000001,"5min":31.033000000000001,"15min":24.866},"min":{"1min":8.2270000000000003,"5min":7.4429999999999996,"15min":7.1870000000000003},"max":{"1min":90.774000000000001,"5min":238.68600000000001,"15min":238.68600000000001},"last":14.224},{"interface":"front","average":{"1min":26.318999999999999,"5min":30.364999999999998,"15min":27.841000000000001},"min":{"1min":10.281000000000001,"5min":9.7170000000000005,"15min":9.5559999999999992},"max":{"1min":89.200000000000003,"5min":174.03800000000001,"15min":242.22800000000001},"last":17.715}]},{"osd":5,"last update":"Sat Jan 27 04:20:28 2024","interfaces":[{"interface":"back","average":{"1min":17.507999999999999,"5min":39.310000000000002,"15min":25.599},"min":{"1min":8.3859999999999992,"5min":7.8310000000000004,"15min":6.665},"max":{"1min":63.085000000000001,"5min":385.50799999999998,"15min":385.50799999999998},"last":11.268000000000001},{"interface":"front","average":{"1min":20.681000000000001,"5min":41.878,"15min":28.116},"min":{"1min":11.789999999999999,"5min":11.631,"15min":6.5839999999999996},"max":{"1min":64.037999999999997,"5min":337.73700000000002,"15min":337.73700000000002},"last":17.248000000000001}]},{"osd":6,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.641999999999999,"5min":39.985999999999997,"15min":31.425000000000001},"min":{"1min":13.096,"5min":10.368,"15min":10.368},"max":{"1min":36.189999999999998,"5min":236.387,"15min":248.78899999999999},"last":13.722},{"interface":"front","average":{"1min":17.960999999999999,"5min":34.942,"15min":28.195},"min":{"1min":7.9359999999999999,"5min":6.4889999999999999,"15min":6.4889999999999999},"max":{"1min":88.625,"5min":252.99700000000001,"15min":381.90800000000002},"last":12.496}]},{"osd":7,"last update":"Sat Jan 27 04:19:59 2024","interfaces":[{"interface":"back","average":{"1min":20.481000000000002,"5min":32.908000000000001,"15min":28.715},"min":{"1min":8.6880000000000006,"5min":8.6880000000000006,"15min":8.4670000000000005},"max":{"1min":56.813000000000002,"5min":232.21600000000001,"15min":236.81299999999999},"last":18.460000000000001},{"interface":"front","average":{"1min":21.489999999999998,"5min":37.271000000000001,"15min":28.934000000000001},"min":{"1min":12.420999999999999,"5min":10.638,"15min":7.2409999999999997},"max":{"1min":34.862000000000002,"5min":249.29499999999999,"15min":249.29499999999999},"last":14.832000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72572,"kb_used_data":5300,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299268,"statfs":{"total":96636764160,"available":96562450432,"internally_reserved":0,"allocated":5427200,"data_stored":2700609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:33 2024","interfaces":[{"interface":"back","average":{"1min":10.587,"5min":18.82,"15min":17.398},"min":{"1min":4.8280000000000003,"5min":4.4569999999999999,"15min":4.3319999999999999},"max":{"1min":20.399000000000001,"5min":203.28899999999999,"15min":278.56900000000002},"last":12.991},{"interface":"front","average":{"1min":9.9540000000000006,"5min":19.867999999999999,"15min":17.433},"min":{"1min":5.6890000000000001,"5min":5.4530000000000003,"15min":4.8129999999999997},"max":{"1min":25.984000000000002,"5min":234.15799999999999,"15min":305.03800000000001},"last":9.1590000000000007}]},{"osd":1,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":31.167000000000002,"5min":26.742000000000001,"15min":23.835999999999999},"min":{"1min":7.3579999999999997,"5min":6.0549999999999997,"15min":6.0549999999999997},"max":{"1min":192.631,"5min":227.602,"15min":304.28300000000002},"last":12.368},{"interface":"front","average":{"1min":28.222000000000001,"5min":23.238,"15min":20.527000000000001},"min":{"1min":6.5300000000000002,"5min":6.5300000000000002,"15min":5.0199999999999996},"max":{"1min":175.59700000000001,"5min":175.59700000000001,"15min":224.27199999999999},"last":15.804}]},{"osd":2,"last update":"Sat Jan 27 04:20:33 2024","interfaces":[{"interface":"back","average":{"1min":18.559999999999999,"5min":28.050000000000001,"15min":28.678000000000001},"min":{"1min":9.5079999999999991,"5min":6.375,"15min":6.375},"max":{"1min":42.564,"5min":239.91900000000001,"15min":311.17200000000003},"last":16.504000000000001},{"interface":"front","average":{"1min":17.369,"5min":25.582000000000001,"15min":25.364999999999998},"min":{"1min":8.3510000000000009,"5min":6.8170000000000002,"15min":6.8170000000000002},"max":{"1min":36.185000000000002,"5min":209.00899999999999,"15min":314.78699999999998},"last":10.324}]},{"osd":3,"last update":"Sat Jan 27 04:20:33 2024","interfaces":[{"interface":"back","average":{"1min":15.244,"5min":21.727,"15min":19.649999999999999},"min":{"1min":8.3409999999999993,"5min":6.7690000000000001,"15min":6.274},"max":{"1min":35.003,"5min":179.31299999999999,"15min":222.04400000000001},"last":6.9400000000000004},{"interface":"front","average":{"1min":17.898,"5min":26.398,"15min":23.744},"min":{"1min":10.417999999999999,"5min":8.0670000000000002,"15min":7.3819999999999997},"max":{"1min":50.088000000000001,"5min":175.44800000000001,"15min":233.72800000000001},"last":10.989000000000001}]},{"osd":4,"last update":"Sat Jan 27 04:20:03 2024","interfaces":[{"interface":"back","average":{"1min":24.469000000000001,"5min":26.721,"15min":22.181000000000001},"min":{"1min":8.5410000000000004,"5min":6.4269999999999996,"15min":5.9059999999999997},"max":{"1min":132.90299999999999,"5min":185.755,"15min":219.21899999999999},"last":8.3339999999999996},{"interface":"front","average":{"1min":26.553000000000001,"5min":30.170999999999999,"15min":25.722000000000001},"min":{"1min":9.5739999999999998,"5min":8.7789999999999999,"15min":7.6689999999999996},"max":{"1min":116.852,"5min":174.81,"15min":174.81},"last":17.227}]},{"osd":6,"last update":"Sat Jan 27 04:20:33 2024","interfaces":[{"interface":"back","average":{"1min":19.373000000000001,"5min":27.314,"15min":26.574000000000002},"min":{"1min":8.907,"5min":8.5150000000000006,"15min":6.9299999999999997},"max":{"1min":40.506,"5min":234.95099999999999,"15min":259.803},"last":9.7300000000000004},{"interface":"front","average":{"1min":22.696000000000002,"5min":29.870999999999999,"15min":28.242000000000001},"min":{"1min":11.374000000000001,"5min":10.73,"15min":8.2859999999999996},"max":{"1min":57.341000000000001,"5min":176.08199999999999,"15min":206.55799999999999},"last":18.408999999999999}]},{"osd":7,"last update":"Sat Jan 27 04:20:14 2024","interfaces":[{"interface":"back","average":{"1min":33.548999999999999,"5min":31.448,"15min":26.803000000000001},"min":{"1min":11.276999999999999,"5min":8.0739999999999998,"15min":8.0739999999999998},"max":{"1min":180.07300000000001,"5min":190.39099999999999,"15min":319.75400000000002},"last":14.647},{"interface":"front","average":{"1min":32.110999999999997,"5min":34.119999999999997,"15min":28.417999999999999},"min":{"1min":10.561999999999999,"5min":9.0800000000000001,"15min":6.6459999999999999},"max":{"1min":217.697,"5min":240.69800000000001,"15min":277.59500000000003},"last":14.019}]}]},{"osd":7,"up_from":13,"seq":55834575297,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72184,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299656,"statfs":{"total":96636764160,"available":96562847744,"internally_reserved":0,"allocated":5029888,"data_stored":2302774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jan 27 04:20:00 2024","interfaces":[{"interface":"back","average":{"1min":10.483000000000001,"5min":21.335000000000001,"15min":15.542999999999999},"min":{"1min":5.649,"5min":4.7370000000000001,"15min":4.1779999999999999},"max":{"1min":24.736999999999998,"5min":162.80799999999999,"15min":162.80799999999999},"last":8.6959999999999997},{"interface":"front","average":{"1min":13.324,"5min":28.123999999999999,"15min":19.064},"min":{"1min":6.5460000000000003,"5min":4.4020000000000001,"15min":4.4020000000000001},"max":{"1min":22.442,"5min":265.62200000000001,"15min":265.62200000000001},"last":13.361000000000001}]},{"osd":1,"last update":"Sat Jan 27 04:20:35 2024","interfaces":[{"interface":"back","average":{"1min":14.890000000000001,"5min":22.393999999999998,"15min":21.088000000000001},"min":{"1min":7.7350000000000003,"5min":6.0899999999999999,"15min":6.0899999999999999},"max":{"1min":54.137999999999998,"5min":133.405,"15min":204.17099999999999},"last":12.199},{"interface":"front","average":{"1min":8.5670000000000002,"5min":16.585000000000001,"15min":18.033000000000001},"min":{"1min":4.7960000000000003,"5min":4.6799999999999997,"15min":4.6799999999999997},"max":{"1min":18.216999999999999,"5min":127.209,"15min":323.48599999999999},"last":7.1900000000000004}]},{"osd":2,"last update":"Sat Jan 27 04:20:31 2024","interfaces":[{"interface":"back","average":{"1min":16.440000000000001,"5min":30.420000000000002,"15min":21.545000000000002},"min":{"1min":7.8289999999999997,"5min":6.6500000000000004,"15min":6.1269999999999998},"max":{"1min":116.20999999999999,"5min":420.65199999999999,"15min":420.65199999999999},"last":8.2680000000000007},{"interface":"front","average":{"1min":15.765000000000001,"5min":28.131,"15min":22.623999999999999},"min":{"1min":6.2460000000000004,"5min":6.2460000000000004,"15min":6.2460000000000004},"max":{"1min":107.611,"5min":453.39100000000002,"15min":453.39100000000002},"last":9.8620000000000001}]},{"osd":3,"last update":"Sat Jan 27 04:20:35 2024","interfaces":[{"interface":"back","average":{"1min":15.795999999999999,"5min":24.626000000000001,"15min":22.324000000000002},"min":{"1min":7.2480000000000002,"5min":7.2480000000000002,"15min":7.2480000000000002},"max":{"1min":49.590000000000003,"5min":140.56200000000001,"15min":226.72300000000001},"last":10.648999999999999},{"interface":"front","average":{"1min":20.256,"5min":22.707000000000001,"15min":21.015999999999998},"min":{"1min":12.484,"5min":6.8810000000000002,"15min":5.7270000000000003},"max":{"1min":67.177999999999997,"5min":139.721,"15min":197.01499999999999},"last":12.807}]},{"osd":4,"last update":"Sat Jan 27 04:20:11 2024","interfaces":[{"interface":"back","average":{"1min":15.734,"5min":29.295000000000002,"15min":22.515000000000001},"min":{"1min":9.2780000000000005,"5min":8.4749999999999996,"15min":8.3200000000000003},"max":{"1min":48.780000000000001,"5min":227.72300000000001,"15min":227.72300000000001},"last":16.300000000000001},{"interface":"front","average":{"1min":18.984000000000002,"5min":32.296999999999997,"15min":24.923999999999999},"min":{"1min":11.102,"5min":9.3059999999999992,"15min":9.3059999999999992},"max":{"1min":55.207000000000001,"5min":230.16,"15min":230.16},"last":15.569000000000001}]},{"osd":5,"last update":"Sat Jan 27 04:20:23 2024","interfaces":[{"interface":"back","average":{"1min":20.927,"5min":39.575000000000003,"15min":27.835000000000001},"min":{"1min":10.061,"5min":8.8070000000000004,"15min":8.8070000000000004},"max":{"1min":113.69799999999999,"5min":276.23500000000001,"15min":276.23500000000001},"last":14.827},{"interface":"front","average":{"1min":23.864000000000001,"5min":36.542000000000002,"15min":29.625},"min":{"1min":12.994999999999999,"5min":10.522,"15min":10.522},"max":{"1min":115.364,"5min":408.774,"15min":408.774},"last":18.047999999999998}]},{"osd":6,"last update":"Sat Jan 27 04:20:35 2024","interfaces":[{"interface":"back","average":{"1min":22.587,"5min":28.809000000000001,"15min":29.207999999999998},"min":{"1min":13.555999999999999,"5min":10.936,"15min":10.936},"max":{"1min":92.113,"5min":137.876,"15min":304.952},"last":17.111999999999998},{"interface":"front","average":{"1min":19.923999999999999,"5min":25.509,"15min":25.170000000000002},"min":{"1min":9.4120000000000008,"5min":7.2199999999999998,"15min":7.2199999999999998},"max":{"1min":96.537000000000006,"5min":196.935,"15min":302.61700000000002},"last":11.664}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-27T04:20:48.706 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.804+0000 7fe2739f0640 10 monclient: get_monmap_and_config 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.804+0000 7fe2739f0640 10 monclient: build_initial_monmap 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.804+0000 7fe2739f0640 10 monclient: monmap: 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T04:20:48.804869+0000 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T04:20:48.804869+0000 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-01-27T04:20:48.805 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.162:6789/0 mon.noname-c-legacy 2024-01-27T04:20:48.806 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-01-27T04:20:48.806 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-01-27T04:20:48.806 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.162:3300/0 mon.noname-c 2024-01-27T04:20:48.806 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.805+0000 7fe2739f0640 1 Processor -- start 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.805+0000 7fe2739f0640 1 -- start start 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.805+0000 7fe2739f0640 10 monclient: init 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe26c0b4680 0x7fe26c142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 10 monclient(hunting): picked mon.noname-c con 0x7fe26c0b4680 addr v2:172.21.15.162:3300/0 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe26c0b4a60 addr v1:172.21.15.162:6789/0 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe26c142eb0 0x7fe26c0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 10 monclient(hunting): picked mon.noname-a con 0x7fe26c142eb0 addr v2:172.21.15.110:3300/0 2024-01-27T04:20:48.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0afbe0 con 0x7fe26c0b4a60 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c142240 con 0x7fe26c142eb0 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c0b0890 con 0x7fe26c0b4680 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe2739f0640 10 monclient(hunting): authenticate will time out at 3732.159103s 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe27178c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe26c0b4680 0x7fe26c142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.806+0000 7fe271f8d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe26c142eb0 0x7fe26c0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe27178c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe26c0b4680 0x7fe26c142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:37678/0 (socket says 172.21.15.110:37678) 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe27178c640 1 -- 172.21.15.110:0/3738555572 learned_addr learned my addr 172.21.15.110:0/3738555572 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe27178c640 10 monclient(hunting): get_auth_request con 0x7fe26c0b4680 auth_method 0 2024-01-27T04:20:48.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe27178c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe27178c640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe27178c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe271f8d640 10 monclient(hunting): get_auth_request con 0x7fe26c142eb0 auth_method 0 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe271f8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe271f8d640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe271f8d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe271f8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe271f8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe271f8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe27178c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe27178c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:48.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.807+0000 7fe27178c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.809+0000 7fe271f8d640 10 monclient(hunting): handle_auth_done global_id 17091 payload 274 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.809+0000 7fe271f8d640 1 -- 172.21.15.110:0/3738555572 >> v2:172.21.15.162:3300/0 conn(0x7fe26c0b4680 msgr2=0x7fe26c142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.809+0000 7fe271f8d640 1 --2- 172.21.15.110:0/3738555572 >> v2:172.21.15.162:3300/0 conn(0x7fe26c0b4680 0x7fe26c142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.809+0000 7fe271f8d640 1 -- 172.21.15.110:0/3738555572 >> v1:172.21.15.162:6789/0 conn(0x7fe26c0b4a60 legacy=0x7fe26c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.809+0000 7fe271f8d640 10 monclient: _finish_hunting 0 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.809+0000 7fe271f8d640 1 monclient: found mon.noname-a 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.809+0000 7fe271f8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.809+0000 7fe271f8d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.809+0000 7fe271f8d640 1 -- 172.21.15.110:0/3738555572 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c0b0b00 con 0x7fe26c142eb0 2024-01-27T04:20:48.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.811+0000 7fe271f8d640 1 --2- 172.21.15.110:0/3738555572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe26c142eb0 0x7fe26c0b1b80 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0b4ee0 tx=0x7fe26402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e983983dd6fdf40 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:48.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 1 -- 172.21.15.110:0/3738555572 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26402cbf0 con 0x7fe26c142eb0 2024-01-27T04:20:48.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 10 monclient: dump: 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:48.816 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 10 monclient: _finish_auth 0 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 10 monclient: _check_auth_tickets 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 1 -- 172.21.15.110:0/3738555572 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26402cd50 con 0x7fe26c142eb0 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 1 -- 172.21.15.110:0/3738555572 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264034660 con 0x7fe26c142eb0 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe263fff640 10 monclient: dump: 2024-01-27T04:20:48.817 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe2739f0640 5 monclient: authenticate success, global_id 17091 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe2739f0640 10 monclient: get_monmap_and_config success 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.814+0000 7fe2739f0640 4 set_mon_vals no callback set 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 10 monclient: shutdown 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:48.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 1 -- 172.21.15.110:0/3738555572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe26c142eb0 msgr2=0x7fe26c0b1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:48.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 1 --2- 172.21.15.110:0/3738555572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe26c142eb0 0x7fe26c0b1b80 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0b4ee0 tx=0x7fe26402be40 comp rx=0 tx=0).stop 2024-01-27T04:20:48.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 1 -- 172.21.15.110:0/3738555572 shutdown_connections 2024-01-27T04:20:48.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 1 --2- 172.21.15.110:0/3738555572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe26c142eb0 0x7fe26c0b1b80 unknown :-1 s=CLOSED pgs=3477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 1 --2- 172.21.15.110:0/3738555572 >> v2:172.21.15.162:3300/0 conn(0x7fe26c0b4680 0x7fe26c142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 1 -- 172.21.15.110:0/3738555572 >> 172.21.15.110:0/3738555572 conn(0x7fe26c066490 msgr2=0x7fe26c0668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:48.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 1 -- 172.21.15.110:0/3738555572 shutdown_connections 2024-01-27T04:20:48.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.815+0000 7fe2739f0640 1 -- 172.21.15.110:0/3738555572 wait complete. 2024-01-27T04:20:48.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.816+0000 7fe2739f0640 10 monclient: build_initial_monmap 2024-01-27T04:20:48.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.816+0000 7fe2739f0640 10 monclient: monmap: 2024-01-27T04:20:48.823 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-01-27T04:20:48.823 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-27T04:20:48.823 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-01-27T04:20:48.823 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-01-27T04:20:48.823 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-01-27T04:20:48.823 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.823 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-01-27T04:20:48.824 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.noname-b 2024-01-27T04:20:48.824 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-01-27T04:20:48.824 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.816+0000 7fe2739f0640 1 librados: starting msgr at 2024-01-27T04:20:48.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.816+0000 7fe2739f0640 1 librados: starting objecter 2024-01-27T04:20:48.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 Processor -- start 2024-01-27T04:20:48.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 -- start start 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 librados: setting wanted keys 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 librados: calling monclient init 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient: init 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient: _reopen_session rank -1 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe26c0b4680 0x7fe26c141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient(hunting): picked mon.noname-a con 0x7fe26c0b4680 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe26c142eb0 0x7fe26c1d8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient(hunting): picked mon.noname-c con 0x7fe26c142eb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-01-27T04:20:48.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe26c1d7a60 0x7fe26c1370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient(hunting): picked mon.noname-b con 0x7fe26c1d7a60 addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c1d8a10 con 0x7fe26c0b4680 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c140510 con 0x7fe26c1d7a60 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient(hunting): start opening mon connection 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c1d7ef0 con 0x7fe26c142eb0 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient(hunting): _renew_subs 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.817+0000 7fe2739f0640 10 monclient(hunting): authenticate will time out at 3732.170610s 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe27178c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe26c0b4680 0x7fe26c141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe270f8b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe26c142eb0 0x7fe26c1d8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe271f8d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe26c1d7a60 0x7fe26c1370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe271f8d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe26c1d7a60 0x7fe26c1370a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.110:37694/0 (socket says 172.21.15.110:37694) 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe271f8d640 1 -- 172.21.15.110:0/983188628 learned_addr learned my addr 172.21.15.110:0/983188628 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-01-27T04:20:48.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe27178c640 10 monclient(hunting): get_auth_request con 0x7fe26c0b4680 auth_method 0 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe27178c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe27178c640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe27178c640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe271f8d640 10 monclient(hunting): get_auth_request con 0x7fe26c1d7a60 auth_method 0 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe271f8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe271f8d640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe271f8d640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe270f8b640 10 monclient(hunting): get_auth_request con 0x7fe26c142eb0 auth_method 0 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe270f8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe270f8b640 10 monclient(hunting): _init_auth method 2 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.818+0000 7fe270f8b640 10 monclient(hunting): _init_auth creating new auth 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.819+0000 7fe271f8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.819+0000 7fe271f8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.819+0000 7fe271f8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.819+0000 7fe27178c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:48.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.819+0000 7fe27178c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.819+0000 7fe27178c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.819+0000 7fe270f8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.819+0000 7fe270f8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.819+0000 7fe270f8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 10 monclient(hunting): handle_auth_done global_id 21013 payload 824 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 1 -- 172.21.15.110:0/983188628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe26c142eb0 msgr2=0x7fe26c1d8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe26c142eb0 0x7fe26c1d8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 1 -- 172.21.15.110:0/983188628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe26c0b4680 msgr2=0x7fe26c141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe26c0b4680 0x7fe26c141290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 10 monclient: _finish_hunting 0 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 1 monclient: found mon.noname-b 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.162:3300/0 2024-01-27T04:20:48.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.821+0000 7fe271f8d640 1 -- 172.21.15.110:0/983188628 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe264009a70 con 0x7fe26c1d7a60 2024-01-27T04:20:48.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.823+0000 7fe271f8d640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe26c1d7a60 0x7fe26c1370a0 secure :-1 s=READY pgs=5110 cs=0 l=1 rev1=1 crypto rx=0x7fe26402c450 tx=0x7fe264004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:48.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.823+0000 7fe261ffb640 10 client.?.objecter ms_handle_connect 0x7fe26c1d7a60 2024-01-27T04:20:48.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.823+0000 7fe261ffb640 10 client.?.objecter resend_mon_ops 2024-01-27T04:20:48.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 1 -- 172.21.15.110:0/983188628 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264040650 con 0x7fe26c1d7a60 2024-01-27T04:20:48.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:48.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-27T04:20:48.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 10 monclient: dump: 2024-01-27T04:20:48.831 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:48.831 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.831 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.832 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.832 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:48.832 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.832 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:48.832 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:48.832 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:48.833 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 10 monclient: _finish_auth 0 2024-01-27T04:20:48.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 10 monclient: _check_auth_tickets 2024-01-27T04:20:48.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-27T04:20:48.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 1 -- 172.21.15.110:0/983188628 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2640407b0 con 0x7fe26c1d7a60 2024-01-27T04:20:48.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-27T04:20:48.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 1 -- 172.21.15.110:0/983188628 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264040b60 con 0x7fe26c1d7a60 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2739f0640 5 monclient: authenticate success, global_id 21013 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2739f0640 10 monclient: _renew_subs 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2739f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c140040 con 0x7fe26c1d7a60 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2637fe640 4 set_mon_vals no callback set 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe261ffb640 10 monclient: dump: 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-01-27T04:20:48.834 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 23f901d8-28a2-405b-9f8e-3221d4911c2c 2024-01-27T04:20:48.835 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.835 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-01-27T03:41:43.032048+0000 2024-01-27T04:20:48.835 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 19 (squid) 2024-01-27T04:20:48.835 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-01-27T04:20:48.835 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-01-27T04:20:48.835 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-01-27T04:20:48.835 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2739f0640 10 client.21013.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2739f0640 10 monclient: _renew_subs 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2739f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c1418e0 con 0x7fe26c1d7a60 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.827+0000 7fe2739f0640 1 librados: init done 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.828+0000 7fe2739f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.828+0000 7fe2739f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.828+0000 7fe2739f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:48.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:48.828+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c074d10 con 0x7fe26c1d7a60 2024-01-27T04:20:49.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.069+0000 7fe261ffb640 1 -- 172.21.15.110:0/983188628 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89250+0+0 (secure 0 0 0) 0x7fe264040cc0 con 0x7fe26c1d7a60 2024-01-27T04:20:49.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.069+0000 7fe261ffb640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe25006c9a0 0x7fe25006ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-27T04:20:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.069+0000 7fe27178c640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe25006c9a0 0x7fe25006ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-27T04:20:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.070+0000 7fe27178c640 10 monclient: get_auth_request con 0x7fe25006c9a0 auth_method 0 2024-01-27T04:20:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.070+0000 7fe27178c640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe25006c9a0 0x7fe25006ee60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe268009bd0 tx=0x7fe268009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T04:20:49.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.070+0000 7fe261ffb640 10 client.21013.objecter ms_handle_connect 0x7fe25006c9a0 2024-01-27T04:20:49.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.072+0000 7fe261ffb640 1 -- 172.21.15.110:0/983188628 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1334..1334 src has 1040..1334) v4 ==== 6377+0+0 (secure 0 0 0) 0x7fe26407b590 con 0x7fe26c1d7a60 2024-01-27T04:20:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.072+0000 7fe261ffb640 10 client.21013.objecter ms_dispatch 0x7fe26c000b90 osd_map(1334..1334 src has 1040..1334) v4 2024-01-27T04:20:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.072+0000 7fe261ffb640 3 client.21013.objecter handle_osd_map got epochs [1334,1334] > 0 2024-01-27T04:20:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.072+0000 7fe261ffb640 3 client.21013.objecter handle_osd_map decoding full epoch 1334 2024-01-27T04:20:49.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.073+0000 7fe261ffb640 20 client.21013.objecter dump_active .. 0 homeless 2024-01-27T04:20:49.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.083+0000 7fe261ffb640 1 -- 172.21.15.110:0/983188628 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180624 (secure 0 0 0) 0x7fe264077830 con 0x7fe26c1d7a60 2024-01-27T04:20:49.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.083+0000 7fe261ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-27T04:20:49.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.083+0000 7fe261ffb640 10 monclient: _finish_command 1 = system:0 2024-01-27T04:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.261+0000 7fe2739f0640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-01-27T04:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.261+0000 7fe2739f0640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-01-27T04:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.261+0000 7fe2739f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.162:3300/0 2024-01-27T04:20:49.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.261+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fe26c0b6b60 con 0x7fe26c1d7a60 2024-01-27T04:20:49.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.425+0000 7fe261ffb640 1 -- 172.21.15.110:0/983188628 <== mon.1 v2:172.21.15.162:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe264077830 con 0x7fe26c1d7a60 2024-01-27T04:20:49.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.425+0000 7fe261ffb640 10 monclient: handle_config config(1 keys) v1 2024-01-27T04:20:49.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.425+0000 7fe2637fe640 4 set_mon_vals no callback set 2024-01-27T04:20:49.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.425+0000 7fe2637fe640 10 set_mon_vals mon_health_to_clog = false 2024-01-27T04:20:49.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.425+0000 7fe261ffb640 1 -- 172.21.15.110:0/983188628 <== mon.1 v2:172.21.15.162:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) v1 ==== 123+0+0 (secure 0 0 0) 0x7fe264009450 con 0x7fe26c1d7a60 2024-01-27T04:20:49.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.425+0000 7fe261ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-01-27T04:20:49.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.425+0000 7fe261ffb640 10 monclient: _finish_command 2 = system:0 2024-01-27T04:20:49.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 10 librados: watch_flush enter 2024-01-27T04:20:49.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 10 librados: watch_flush exit 2024-01-27T04:20:49.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 20 client.21013.objecter shutdown clearing up homeless session... 2024-01-27T04:20:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 10 client.21013.objecter successfully canceled tick 2024-01-27T04:20:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe25006c9a0 msgr2=0x7fe25006ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe25006c9a0 0x7fe25006ee60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe268009bd0 tx=0x7fe268009240 comp rx=0 tx=0).stop 2024-01-27T04:20:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 10 monclient: shutdown 2024-01-27T04:20:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-27T04:20:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe26c1d7a60 msgr2=0x7fe26c1370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T04:20:49.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe26c1d7a60 0x7fe26c1370a0 secure :-1 s=READY pgs=5110 cs=0 l=1 rev1=1 crypto rx=0x7fe26402c450 tx=0x7fe264004a40 comp rx=0 tx=0).stop 2024-01-27T04:20:49.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 shutdown_connections 2024-01-27T04:20:49.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:6800/3672294946,v1:172.21.15.162:6801/3672294946] conn(0x7fe25006c9a0 0x7fe25006ee60 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:49.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe26c142eb0 0x7fe26c1d8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:49.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe26c1d7a60 0x7fe26c1370a0 unknown :-1 s=CLOSED pgs=5110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:49.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 --2- 172.21.15.110:0/983188628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe26c0b4680 0x7fe26c141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-27T04:20:49.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 >> 172.21.15.110:0/983188628 conn(0x7fe26c066490 msgr2=0x7fe26c13b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T04:20:49.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.428+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 shutdown_connections 2024-01-27T04:20:49.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.429+0000 7fe2739f0640 1 -- 172.21.15.110:0/983188628 wait complete. 2024-01-27T04:20:49.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-01-27T04:20:49.429+0000 7fe2739f0640 1 librados: shutdown 2024-01-27T04:20:49.442 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-27T04:20:49.443 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-27T04:20:49.443 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-27T04:20:49.443 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:20:49.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T04:20:49.447+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 34860) UID: 0 2024-01-27T04:20:49.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-01-27T04:20:49.449+0000 b47f640 -1 osd.0 1334 *** Got signal Terminated *** 2024-01-27T04:20:53.041 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:38.997 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:53.044 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:39.001 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:53.058 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:39.014 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:53.087 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:39.033 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:53.091 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:39.036 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:53.091 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:39.047 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:53.092 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:39.048 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:53.105 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:39.061 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:53.106 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:39.051 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:20:55.545 INFO:tasks.ceph.osd.0:Stopped 2024-01-27T04:20:55.546 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-27T04:20:55.546 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:20:55.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T04:20:55.549+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 34865) UID: 0 2024-01-27T04:20:55.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-01-27T04:20:55.550+0000 b47f640 -1 osd.1 1336 *** Got signal Terminated *** 2024-01-27T04:21:03.211 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:49.164 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:03.212 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:49.168 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:03.227 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:49.183 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:03.255 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:49.200 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:03.257 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:49.202 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:03.258 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:49.214 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:03.262 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:49.218 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:03.272 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:49.217 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:03.279 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:49.235 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:08.100 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:54.056 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:08.102 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:54.058 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:08.111 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:38:54.067 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:08.147 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:54.093 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:08.148 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:54.104 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:08.149 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:54.095 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:08.150 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:54.106 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:08.162 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:38:54.118 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:08.164 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:38:54.110 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:23.086 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:09.042 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:23.151 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:09.107 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:23.170 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:09.126 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:23.202 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:09.158 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:23.205 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:09.150 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:23.208 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:09.164 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:23.209 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:09.155 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:23.224 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:09.180 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:23.227 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:09.173 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:28.100 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:14.056 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:28.152 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:14.107 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:28.165 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:14.121 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:28.198 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:14.143 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:28.199 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:14.156 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:28.201 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:14.146 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:28.202 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:14.158 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:28.213 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:14.169 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:28.214 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:14.159 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:43.181 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:29.137 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:43.185 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:29.141 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:43.202 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:29.158 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:43.236 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:29.181 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:43.242 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:29.188 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:43.256 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:29.212 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:43.257 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:29.203 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:43.261 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:29.217 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:43.279 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:29.235 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:48.197 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:34.153 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:48.200 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:34.156 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:48.214 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:34.170 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:48.248 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:34.193 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:48.249 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:34.205 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:48.251 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:34.207 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:48.255 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:34.201 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:48.268 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:34.224 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:48.270 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:34.215 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:21:53.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-27T04:21:56.561 INFO:tasks.ceph.osd.1:Stopped 2024-01-27T04:21:56.562 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-27T04:21:56.562 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:21:56.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T04:21:56.565+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 34872) UID: 0 2024-01-27T04:21:56.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-01-27T04:21:56.566+0000 b47f640 -1 osd.2 1340 *** Got signal Terminated *** 2024-01-27T04:21:59.266 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-01-27T04:22:03.292 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:49.248 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:03.341 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:49.297 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:03.341 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:49.287 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:03.342 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:49.288 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:03.343 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:49.299 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:03.359 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:49.304 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:03.366 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:49.322 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:03.366 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:49.322 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:03.386 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:49.343 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:08.261 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:54.217 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:08.266 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:54.222 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:08.278 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:39:54.234 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:08.315 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:54.271 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:08.316 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:54.272 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:08.319 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:54.264 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:08.321 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:54.267 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:08.329 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:39:54.285 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:08.338 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:39:54.283 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:23.308 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:09.263 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:23.309 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:09.265 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:23.323 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:09.279 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:23.362 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:09.307 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:23.364 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:09.309 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:23.369 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:09.325 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:23.373 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:09.329 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:23.379 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:09.324 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:23.390 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:09.346 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:28.317 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:14.273 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:28.322 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:14.278 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:28.334 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:14.290 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:28.372 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:14.327 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:28.373 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:14.329 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:28.373 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:14.319 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:28.377 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:14.322 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:28.388 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:14.344 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:28.391 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:14.337 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:43.348 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:29.304 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:43.398 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:29.354 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:43.398 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:29.355 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:43.399 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:29.344 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:43.400 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:29.345 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:43.400 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:29.356 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:43.416 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:29.361 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:43.418 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:29.374 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:43.427 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:29.383 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:48.406 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:34.361 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:48.411 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:34.367 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:48.424 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:34.380 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:48.457 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:34.413 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:48.458 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:34.414 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:48.467 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:34.413 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:48.468 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:34.414 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:48.470 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:34.427 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:22:48.482 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:34.427 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:03.454 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:49.410 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:03.458 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:49.414 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:03.504 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:49.460 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:03.507 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:49.464 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:03.511 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:49.467 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:03.512 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:49.458 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:03.514 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:49.460 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:03.533 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:49.478 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:03.539 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:49.495 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:08.426 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:54.382 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:08.473 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:54.429 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:08.486 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:40:54.442 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:08.521 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:54.466 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:08.521 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:54.477 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:08.524 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:54.469 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:08.527 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:54.483 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:08.537 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:40:54.482 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:08.542 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:40:54.499 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:15.881 INFO:tasks.ceph.osd.2:Stopped 2024-01-27T04:23:15.882 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-27T04:23:15.882 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:23:15.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-27T04:23:15.885+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 34874) UID: 0 2024-01-27T04:23:15.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-01-27T04:23:15.886+0000 b47f640 -1 osd.3 1344 *** Got signal Terminated *** 2024-01-27T04:23:21.984 INFO:tasks.ceph.osd.3:Stopped 2024-01-27T04:23:21.984 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-27T04:23:21.984 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:23:21.997 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T04:23:21.987+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 34661) UID: 0 2024-01-27T04:23:21.998 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-01-27T04:23:21.988+0000 b47f640 -1 osd.4 1346 *** Got signal Terminated *** 2024-01-27T04:23:23.627 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:09.583 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:23.633 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:09.589 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:23.651 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:09.607 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:23.679 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:09.624 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:23.680 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:09.625 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:23.685 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:09.641 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:23.686 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:09.642 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:23.699 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:09.643 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:23.704 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:09.660 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:28.528 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:14.483 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:28.533 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:14.487 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:28.543 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:14.499 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:28.578 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:14.524 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:28.581 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:14.537 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:28.582 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:14.527 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:28.583 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:14.539 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:28.594 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:14.538 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:28.594 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:14.549 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:43.530 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:29.485 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:43.576 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:29.521 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:43.578 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:29.524 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:43.580 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:29.536 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:43.597 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:29.542 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:43.603 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:29.557 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:43.647 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:29.603 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:43.649 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:29.604 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:43.664 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:29.619 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:44.358 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:30.303 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:44.359 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:30.304 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:44.371 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:30.317 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:44.417 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:30.373 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:44.418 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:30.375 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:44.433 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:30.388 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:44.438 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:30.394 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:44.439 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:30.395 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:44.452 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:30.408 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:48.590 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:34.546 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:48.594 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:34.550 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:48.607 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:34.563 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:48.639 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:34.584 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:48.641 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:34.587 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:48.643 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:34.599 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:48.645 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:34.601 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:48.652 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:34.598 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:23:48.657 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:34.614 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:03.636 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:49.592 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:03.638 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:49.594 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:03.654 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:49.610 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:03.686 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:49.642 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:03.688 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:49.634 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:03.689 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:49.645 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:03.689 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:49.635 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:03.703 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:49.648 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:03.705 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:49.661 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:08.651 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:54.607 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:08.653 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:54.609 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:08.667 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:41:54.623 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:08.702 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:54.647 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:08.702 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:54.658 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:08.703 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:54.649 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:08.705 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:54.661 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:08.714 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:41:54.659 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:08.716 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:41:54.672 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:22.999 INFO:tasks.ceph.osd.4:Stopped 2024-01-27T04:24:22.999 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-27T04:24:22.999 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:24:23.016 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T04:24:23.002+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 34664) UID: 0 2024-01-27T04:24:23.017 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-01-27T04:24:23.003+0000 b47f640 -1 osd.5 1348 *** Got signal Terminated *** 2024-01-27T04:24:23.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-27T04:24:23.714 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:09.670 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:23.717 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:09.673 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:23.733 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:09.689 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:23.764 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:09.720 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:23.765 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:09.710 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:23.767 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:09.724 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:23.768 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:09.713 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:23.783 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:09.739 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:23.783 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:09.729 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:28.733 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:14.689 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:28.737 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:14.693 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:28.754 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:14.710 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:28.788 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:14.733 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:28.788 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:14.744 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:28.789 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:14.745 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:28.792 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:14.737 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:28.801 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:14.757 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:28.806 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:14.751 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:29.101 INFO:tasks.ceph.osd.5:Stopped 2024-01-27T04:24:29.101 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-27T04:24:29.101 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:24:29.114 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T04:24:29.104+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 34666) UID: 0 2024-01-27T04:24:29.116 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-01-27T04:24:29.105+0000 b47f640 -1 osd.6 1350 *** Got signal Terminated *** 2024-01-27T04:24:29.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-01-27T04:24:29.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-01-27T04:24:34.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-01-27T04:24:43.773 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:29.729 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:43.778 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:29.734 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:43.800 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:29.756 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:43.825 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:29.771 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:43.828 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:29.773 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:43.836 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:29.792 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:43.837 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:29.793 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:43.842 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:29.788 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:43.852 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:29.808 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:48.789 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:34.745 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:48.793 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:34.749 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:48.806 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:34.763 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:48.835 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:34.781 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:48.840 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:34.786 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:48.842 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:34.798 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:48.845 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:34.801 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:48.851 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:34.797 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:24:48.859 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:34.815 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:03.784 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:49.740 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:03.832 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:49.788 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:03.832 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:49.778 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:03.838 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:49.783 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:03.853 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:49.799 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:03.854 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:49.810 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:03.882 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:49.838 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:03.885 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:49.841 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:03.906 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:49.862 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:08.841 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:54.797 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:08.845 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:54.801 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:08.857 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:42:54.814 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:08.891 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:54.836 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:08.892 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:54.848 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:08.893 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:54.839 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:08.896 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:54.852 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:08.905 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:42:54.851 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:08.909 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:42:54.865 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:23.846 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:09.801 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:23.896 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:09.852 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:23.907 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:09.852 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:23.914 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:09.869 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:23.918 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:09.864 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:23.933 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:09.878 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:23.948 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:09.904 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:23.951 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:09.908 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:23.967 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:09.923 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:33.918 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:19.873 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:33.922 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:19.878 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:33.934 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:19.890 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:33.968 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:19.924 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:33.970 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:19.926 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:33.974 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:19.920 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:33.977 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:19.922 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:33.983 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:19.939 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:33.989 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:19.935 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:43.945 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:29.901 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:43.947 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:29.903 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:43.964 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:29.920 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:43.993 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:29.939 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:43.994 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:29.950 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:43.997 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:29.953 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:44.009 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:29.954 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:44.016 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:29.972 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:44.023 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:29.969 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:53.981 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:39.937 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:53.982 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:39.938 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:53.994 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:39.950 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:54.025 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:39.970 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:54.029 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:39.986 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:54.032 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:39.988 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:54.041 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:39.987 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:54.044 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:40.000 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:25:54.054 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:39.999 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:09.018 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:54.974 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:09.023 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:54.979 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:09.041 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:54.997 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:09.071 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:55.017 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:09.072 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:55.018 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:09.074 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:55.030 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:09.076 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:55.032 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:09.088 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:43:55.034 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:09.093 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:43:55.049 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:14.034 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:59.990 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:14.039 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:43:59.995 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:14.051 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:44:00.007 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:14.083 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:44:00.029 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:14.086 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:44:00.042 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:14.087 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:44:00.033 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:14.090 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:44:00.046 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:14.097 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:44:00.043 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:14.101 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:44:00.057 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:17.450 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:17.444+0000 ea50640 -1 osd.7 1350 heartbeat_check: no reply from 172.21.15.162:6814 osd.6 since back 2024-01-27T04:24:28.414731+0000 front 2024-01-27T04:24:28.416304+0000 (oldest deadline 2024-01-27T04:26:16.519868+0000) 2024-01-27T04:26:18.495 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:18.492+0000 ea50640 -1 osd.7 1350 heartbeat_check: no reply from 172.21.15.162:6814 osd.6 since back 2024-01-27T04:24:28.414731+0000 front 2024-01-27T04:24:28.416304+0000 (oldest deadline 2024-01-27T04:26:16.519868+0000) 2024-01-27T04:26:19.457 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:19.454+0000 ea50640 -1 osd.7 1350 heartbeat_check: no reply from 172.21.15.162:6814 osd.6 since back 2024-01-27T04:24:28.414731+0000 front 2024-01-27T04:24:28.416304+0000 (oldest deadline 2024-01-27T04:26:16.519868+0000) 2024-01-27T04:26:20.503 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:20.497+0000 ea50640 -1 osd.7 1350 heartbeat_check: no reply from 172.21.15.162:6814 osd.6 since back 2024-01-27T04:24:28.414731+0000 front 2024-01-27T04:24:28.416304+0000 (oldest deadline 2024-01-27T04:26:16.519868+0000) 2024-01-27T04:26:21.517 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:21.515+0000 ea50640 -1 osd.7 1350 heartbeat_check: no reply from 172.21.15.162:6814 osd.6 since back 2024-01-27T04:24:28.414731+0000 front 2024-01-27T04:24:28.416304+0000 (oldest deadline 2024-01-27T04:26:16.519868+0000) 2024-01-27T04:26:22.512 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:22.509+0000 ea50640 -1 osd.7 1350 heartbeat_check: no reply from 172.21.15.162:6814 osd.6 since back 2024-01-27T04:24:28.414731+0000 front 2024-01-27T04:24:28.416304+0000 (oldest deadline 2024-01-27T04:26:16.519868+0000) 2024-01-27T04:26:23.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-01-27T04:26:23.562 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:23.560+0000 ea50640 -1 osd.7 1350 heartbeat_check: no reply from 172.21.15.162:6814 osd.6 since back 2024-01-27T04:24:28.414731+0000 front 2024-01-27T04:24:28.416304+0000 (oldest deadline 2024-01-27T04:26:16.519868+0000) 2024-01-27T04:26:24.514 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:24.512+0000 ea50640 -1 osd.7 1350 heartbeat_check: no reply from 172.21.15.162:6814 osd.6 since back 2024-01-27T04:24:28.414731+0000 front 2024-01-27T04:24:28.416304+0000 (oldest deadline 2024-01-27T04:26:16.519868+0000) 2024-01-27T04:26:25.029 INFO:tasks.ceph.osd.6:Stopped 2024-01-27T04:26:25.029 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-27T04:26:25.030 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:26:25.045 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:25.033+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 34668) UID: 0 2024-01-27T04:26:25.046 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-01-27T04:26:25.035+0000 b47f640 -1 osd.7 1350 *** Got signal Terminated *** 2024-01-27T04:26:28.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-01-27T04:26:29.197 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:44:15.153 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:29.202 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:44:15.158 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:29.221 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:44:15.177 34563== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:29.248 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:44:15.194 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:29.249 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:44:15.205 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:29.251 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:44:15.197 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:29.251 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:44:15.208 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:29.268 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:44:15.213 34051== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:29.271 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:44:15.227 34567== Warning: unimplemented fcntl command: 1036 2024-01-27T04:26:31.131 INFO:tasks.ceph.osd.7:Stopped 2024-01-27T04:26:31.132 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-27T04:26:31.132 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-27T04:26:31.132 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:26:31.157 INFO:tasks.ceph.mgr.y:Stopped 2024-01-27T04:26:31.158 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-27T04:26:31.158 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:26:31.159 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T04:26:31.158+0000 7f872f0cc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34049) UID: 0 2024-01-27T04:26:31.159 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-01-27T04:26:31.158+0000 7f872f0cc640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-01-27T04:26:31.185 INFO:tasks.ceph.mgr.x:Stopped 2024-01-27T04:26:31.185 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-27T04:26:31.185 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-27T04:26:31.186 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:26:31.212 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-01-27T04:26:31.201+0000 9f78640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34535) UID: 0 2024-01-27T04:26:31.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-01-27T04:26:31.203+0000 9f78640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-01-27T04:26:31.916 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:44:17.872 34563== 2024-01-27T04:26:31.917 INFO:tasks.ceph.mon.a.smithi110.stderr:==00:00:44:17.872 34563== Exit program on first error (--exit-on-first-error=yes) 2024-01-27T04:26:31.941 INFO:tasks.ceph.mon.a.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-01-27T04:26:33.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-01-27T04:26:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-27T04:26:37.287 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-01-27T04:26:37.288 INFO:tasks.ceph.mon.a:Stopped 2024-01-27T04:26:37.288 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-27T04:26:37.288 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:26:37.304 INFO:tasks.ceph.mon.c.smithi110.stderr:2024-01-27T04:26:37.292+0000 9f78640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34537) UID: 0 2024-01-27T04:26:37.304 INFO:tasks.ceph.mon.c.smithi110.stderr:2024-01-27T04:26:37.293+0000 9f78640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-01-27T04:26:38.109 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:44:24.065 34567== 2024-01-27T04:26:38.109 INFO:tasks.ceph.mon.c.smithi110.stderr:==00:00:44:24.065 34567== Exit program on first error (--exit-on-first-error=yes) 2024-01-27T04:26:38.129 INFO:tasks.ceph.mon.c.smithi110.stderr:daemon-helper: command failed with exit status 42 2024-01-27T04:26:39.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-01-27T04:26:39.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-01-27T04:26:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-27T04:26:43.391 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-01-27T04:26:43.392 INFO:tasks.ceph.mon.c:Stopped 2024-01-27T04:26:43.392 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-27T04:26:43.392 INFO:teuthology.orchestra.run:waiting for 300 2024-01-27T04:26:43.408 INFO:tasks.ceph.mon.b.smithi162.stderr:2024-01-27T04:26:43.396+0000 9f78640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34047) UID: 0 2024-01-27T04:26:43.408 INFO:tasks.ceph.mon.b.smithi162.stderr:2024-01-27T04:26:43.398+0000 9f78640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-01-27T04:26:44.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-01-27T04:27:21.124 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:45:07.069 34051== 2024-01-27T04:27:21.124 INFO:tasks.ceph.mon.b.smithi162.stderr:==00:00:45:07.069 34051== Exit program on first error (--exit-on-first-error=yes) 2024-01-27T04:27:21.143 INFO:tasks.ceph.mon.b.smithi162.stderr:daemon-helper: command failed with exit status 42 2024-01-27T04:27:24.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-01-27T04:27:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-27T04:27:26.102 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-01-27T04:27:26.102 INFO:tasks.ceph.mon.b:Stopped 2024-01-27T04:27:26.103 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-27T04:27:26.103 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-01-27T04:27:26.176 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T04:27:26.177 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-27T04:27:26.237 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T04:27:26.237 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-27T04:27:26.292 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T04:27:26.292 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-27T04:27:26.362 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T04:27:26.362 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-27T04:27:26.443 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T04:27:26.443 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-27T04:27:26.581 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T04:27:26.581 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-27T04:27:26.653 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T04:27:26.654 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-27T04:27:26.711 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T04:27:26.711 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-27T04:27:26.768 INFO:tasks.ceph:Archiving mon data... 2024-01-27T04:27:26.770 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/data/mon.a.tgz 2024-01-27T04:27:26.770 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-01-27T04:27:26.785 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.GYS9wJtExu 2024-01-27T04:27:26.786 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GYS9wJtExu 2024-01-27T04:27:26.961 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.GYS9wJtExu 2024-01-27T04:27:27.046 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.GYS9wJtExu is 548KB 2024-01-27T04:27:27.146 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.GYS9wJtExu 2024-01-27T04:27:27.161 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/data/mon.c.tgz 2024-01-27T04:27:27.161 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-01-27T04:27:27.216 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.WIDwcJCQmw 2024-01-27T04:27:27.216 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.WIDwcJCQmw 2024-01-27T04:27:27.375 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.WIDwcJCQmw 2024-01-27T04:27:27.460 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.WIDwcJCQmw is 549KB 2024-01-27T04:27:27.557 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.WIDwcJCQmw 2024-01-27T04:27:27.571 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/data/mon.b.tgz 2024-01-27T04:27:27.571 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-01-27T04:27:27.589 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.pUrmzGLIF1 2024-01-27T04:27:27.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.pUrmzGLIF1 2024-01-27T04:27:27.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.pUrmzGLIF1 2024-01-27T04:27:27.863 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.pUrmzGLIF1 is 549KB 2024-01-27T04:27:27.962 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.pUrmzGLIF1 2024-01-27T04:27:27.977 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-27T04:27:27.977 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-27T04:27:27.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-27T04:27:28.305 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-27T04:27:28.542 INFO:tasks.ceph:Archiving crash dumps... 2024-01-27T04:27:28.544 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/remote/smithi110/crash 2024-01-27T04:27:28.545 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-27T04:27:28.573 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/remote/smithi162/crash 2024-01-27T04:27:28.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-27T04:27:28.600 INFO:tasks.ceph:Compressing logs... 2024-01-27T04:27:28.600 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-27T04:27:28.617 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-27T04:27:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38528.log 2024-01-27T04:27:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41365.log 2024-01-27T04:27:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34577.log 2024-01-27T04:27:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41082.log 2024-01-27T04:27:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34577.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38528.log.gz 2024-01-27T04:27:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39634.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41082.log: /var/log/ceph/ceph-client.admin.39634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107926.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.34577.log.gz 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41165.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39634.log.gz 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63875.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38932.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107926.log.gz 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.63875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41293.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40917.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40169.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63875.log.gz 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38932.log.gz 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107884.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40917.log: /var/log/ceph/ceph-client.admin.40169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40917.log.gz 2024-01-27T04:27:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107590.log 2024-01-27T04:27:28.648 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107884.log.gz 2024-01-27T04:27:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41013.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.41082.log.gz 2024-01-27T04:27:28.651 INFO:teuthology.orchestra.run.smithi110.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2024-01-27T04:27:28.652 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108178.log 2024-01-27T04:27:28.652 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38931.log 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108220.log 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107590.log.gz 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35924.log 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108178.log.gz 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108220.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40169.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40685.log 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38931.log.gz 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.35924.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108220.log.gz 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62400.log 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40685.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35924.log.gz 2024-01-27T04:27:28.674 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39504.log 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.62400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39564.log 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39564.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39504.log.gz 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41293.log.gz 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39564.log.gzgzip -5 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40246.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/mon.c.log: gzip 91.3% -- replaced with /var/log/ceph/ceph-client.admin.62400.log.gz 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/valgrind/osd.2.log: -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-01-27T04:27:28.675 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.1.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39242.log 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38928.log 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39842.log 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39242.log.gz 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108346.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38928.log.gz 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107674.log 2024-01-27T04:27:28.676 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81311.log 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39842.log.gz 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107674.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108346.log.gz 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107674.log.gz 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63559.log 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39605.log 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.63559.log: /var/log/ceph/ceph-client.admin.81311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63874.log 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40685.log.gz 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40839.log 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.63874.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39605.log.gz 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.63559.log.gz 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.81311.log.gz 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34879.log 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41230.log 2024-01-27T04:27:28.677 INFO:teuthology.orchestra.run.smithi110.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.63874.log.gz 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38750.log 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41230.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34879.log.gz 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.40712.log 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90884.log 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40712.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40874.log 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38750.log.gz 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.90884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41092.log 91.6% 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40839.log.gz 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40874.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.90884.log.gz 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39524.log 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40364.log 2024-01-27T04:27:28.678 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39524.log: /var/log/ceph/ceph-client.admin.41092.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39524.log.gz 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.41092.log.gz 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81312.log 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39197.log 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.40874.log.gz 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.81312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41112.log 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39197.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39197.log.gz 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39208.log 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41112.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.81312.log.gz 2024-01-27T04:27:28.689 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73891.log 2024-01-27T04:27:28.690 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39208.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39208.log.gz 2024-01-27T04:27:28.690 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39464.log 2024-01-27T04:27:28.690 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.73891.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.73891.log.gz 2024-01-27T04:27:28.690 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108388.log 2024-01-27T04:27:28.690 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39464.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39464.log.gz 2024-01-27T04:27:28.690 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73892.log 2024-01-27T04:27:28.690 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108388.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108388.log.gz 2024-01-27T04:27:28.690 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41001.log 2024-01-27T04:27:28.690 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.73892.log: 96.6% -- replaced with /var/log/ceph/ceph-client.admin.41165.log.gz 2024-01-27T04:27:28.696 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-27T04:27:28.696 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-01-27T04:27:28.696 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34284.log 2024-01-27T04:27:28.696 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34456.log 2024-01-27T04:27:28.696 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.34284.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-01-27T04:27:28.697 INFO:teuthology.orchestra.run.smithi162.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-01-27T04:27:28.697 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-01-27T04:27:28.697 INFO:teuthology.orchestra.run.smithi162.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34284.log.gz 2024-01-27T04:27:28.697 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.34456.log: /var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-01-27T04:27:28.697 INFO:teuthology.orchestra.run.smithi162.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34456.log.gz 2024-01-27T04:27:28.698 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/osd.5.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-01-27T04:27:28.698 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-01-27T04:27:28.698 INFO:teuthology.orchestra.run.smithi162.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-01-27T04:27:28.698 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-01-27T04:27:28.698 INFO:teuthology.orchestra.run.smithi162.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-01-27T04:27:28.698 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34364.log 2024-01-27T04:27:28.698 INFO:teuthology.orchestra.run.smithi162.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-01-27T04:27:28.699 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34405.log 2024-01-27T04:27:28.699 INFO:teuthology.orchestra.run.smithi162.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-01-27T04:27:28.699 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.34364.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-27T04:27:28.699 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.34405.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-27T04:27:28.699 INFO:teuthology.orchestra.run.smithi162.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34364.log.gz 2024-01-27T04:27:28.699 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34324.log 2024-01-27T04:27:28.700 INFO:teuthology.orchestra.run.smithi162.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34405.log.gz 2024-01-27T04:27:28.700 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-27T04:27:28.700 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.34324.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-27T04:27:28.700 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-27T04:27:28.700 INFO:teuthology.orchestra.run.smithi162.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34324.log.gz 2024-01-27T04:27:28.700 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-27T04:27:28.701 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-27T04:27:28.701 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34244.log 2024-01-27T04:27:28.701 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-mon.b.log: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-27T04:27:28.701 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34204.log 2024-01-27T04:27:28.701 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.34244.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34244.log.gz 2024-01-27T04:27:28.703 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.34204.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34204.log.gz 2024-01-27T04:27:28.703 INFO:teuthology.orchestra.run.smithi162.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-27T04:27:28.734 INFO:teuthology.orchestra.run.smithi110.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.73892.log.gz 2024-01-27T04:27:28.734 INFO:teuthology.orchestra.run.smithi110.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41365.log.gz 2024-01-27T04:27:28.734 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40530.log 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108010.log 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36945.log 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108010.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108010.log.gz 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41112.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40469.log 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.36945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107800.log 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107800.log: 92.1% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36945.log.gz 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41001.log.gz 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39800.log 2024-01-27T04:27:28.735 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107800.log.gz 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35045.log 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr: 92.3%/var/log/ceph/ceph-client.admin.35045.log: -- replaced with /var/log/ceph/ceph-client.admin.41230.log.gz 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39800.log.gz 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35045.log.gz 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39170.log 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40252.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.40530.log.gz 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38181.log 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40767.log 2024-01-27T04:27:28.736 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39170.log: /var/log/ceph/ceph-client.admin.38181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39323.log 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40767.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39170.log.gz 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38181.log.gz 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39323.log: -- /var/log/ceph/ceph-client.admin.41179.log 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107758.log 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39323.log.gz 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40481.log 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107758.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.107758.log.gz 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81317.log 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40481.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40767.log.gz 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38666.log 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.81317.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.81317.log.gz 2024-01-27T04:27:28.737 INFO:teuthology.orchestra.run.smithi110.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.41179.log.gz 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39758.log 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38708.log 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39758.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28726.log 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38666.log.gz 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38708.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39758.log.gz 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.108262.log 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.tmp-client.admin.28726.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28726.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41213.log 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34906.log 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38708.log.gz 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39185.log 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108262.log.gz 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.34906.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34906.log.gz 2024-01-27T04:27:28.738 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41266.log 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63560.log 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39185.log: /var/log/ceph/ceph-client.admin.41266.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39185.log.gz 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38415.log 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.63560.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63560.log.gz 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38468.log 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38415.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38415.log.gz 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40510.log 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38468.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41266.log.gz 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38468.log.gz 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107506.log 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41050.log 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40510.log: /var/log/ceph/ceph-client.admin.107506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108094.log 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41050.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107506.log.gz 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.41050.log.gz 2024-01-27T04:27:28.739 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40510.log.gz 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108094.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62401.log 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2%/var/log/ceph/ceph-osd.3.log: -- replaced with /var/log/ceph/ceph-client.admin.108094.log.gz 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38930.log 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.62401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107716.log 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38930.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38930.log.gz 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.62401.log.gz 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41016.log 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40250.log 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41016.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107716.log.gz 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90885.log 2024-01-27T04:27:28.740 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40250.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41016.log.gz 2024-01-27T04:27:28.742 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41000.log 2024-01-27T04:27:28.742 INFO:teuthology.orchestra.run.smithi110.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41213.log.gz 2024-01-27T04:27:28.742 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38926.log 2024-01-27T04:27:28.742 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.90885.log: /var/log/ceph/ceph-client.admin.41000.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.90885.log.gz 2024-01-27T04:27:28.744 INFO:teuthology.orchestra.run.smithi110.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.41000.log.gz 2024-01-27T04:27:28.744 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107548.log 2024-01-27T04:27:28.745 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73890.log 2024-01-27T04:27:28.746 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107548.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38926.log.gz 2024-01-27T04:27:28.746 INFO:teuthology.orchestra.run.smithi110.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107548.log.gz 2024-01-27T04:27:28.746 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38624.log 2024-01-27T04:27:28.746 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.73890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107842.log 2024-01-27T04:27:28.747 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38624.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38624.log.gz 2024-01-27T04:27:28.748 INFO:teuthology.orchestra.run.smithi110.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.73890.log.gz 2024-01-27T04:27:28.749 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40638.log 2024-01-27T04:27:28.749 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63562.log 2024-01-27T04:27:28.751 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40638.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107842.log.gz 2024-01-27T04:27:28.751 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-27T04:27:28.753 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.63562.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.63562.log.gz 2024-01-27T04:27:28.754 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90882.log 2024-01-27T04:27:28.759 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mgr.y.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40638.log.gz 2024-01-27T04:27:28.766 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40157.log 2024-01-27T04:27:28.769 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.90882.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.90882.log.gz 2024-01-27T04:27:28.771 INFO:teuthology.orchestra.run.smithi110.stderr: 94.3% -- replaced with /var/log/ceph/ceph-client.admin.40364.log.gz 2024-01-27T04:27:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40234.log 2024-01-27T04:27:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107632.log 2024-01-27T04:27:28.784 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40234.log: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-27T04:27:28.785 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40187.log 2024-01-27T04:27:28.786 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107632.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107632.log.gz 2024-01-27T04:27:28.787 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41333.log 2024-01-27T04:27:28.797 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40187.log: 92.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2024-01-27T04:27:28.802 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41333.log: -- replaced with /var/log/ceph/ceph-client.admin.40187.log.gz 2024-01-27T04:27:28.809 INFO:teuthology.orchestra.run.smithi110.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41333.log.gz 2024-01-27T04:27:28.813 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-27T04:27:28.824 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40634.log: 92.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108052.log 2024-01-27T04:27:28.832 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2024-01-27T04:27:28.836 INFO:teuthology.orchestra.run.smithi110.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.40234.log.gz 2024-01-27T04:27:28.846 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41054.log 2024-01-27T04:27:28.848 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108052.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108052.log.gz 2024-01-27T04:27:28.848 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-27T04:27:28.848 INFO:teuthology.orchestra.run.smithi110.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40157.log.gz 2024-01-27T04:27:28.849 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41054.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-27T04:27:28.849 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.41054.log.gz 2024-01-27T04:27:28.850 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81318.log 2024-01-27T04:27:28.863 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108136.log 2024-01-27T04:27:28.865 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.81318.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-27T04:27:28.867 INFO:teuthology.orchestra.run.smithi110.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.81318.log.gz 2024-01-27T04:27:28.867 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81314.log 2024-01-27T04:27:28.868 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108136.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108136.log.gz 2024-01-27T04:27:28.868 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62399.log 2024-01-27T04:27:28.872 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.81314.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.81314.log.gz 2024-01-27T04:27:28.872 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38929.log 2024-01-27T04:27:28.876 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.62399.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.62399.log.gz 2024-01-27T04:27:28.876 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40595.log 2024-01-27T04:27:28.877 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38929.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38929.log.gz 2024-01-27T04:27:28.877 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38584.log 2024-01-27T04:27:28.888 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40595.log: 92.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40783.log 2024-01-27T04:27:28.889 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38584.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38584.log.gz 2024-01-27T04:27:28.890 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40595.log.gz 2024-01-27T04:27:28.890 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108304.log 2024-01-27T04:27:28.909 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2024-01-27T04:27:28.910 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108304.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108304.log.gz 2024-01-27T04:27:28.916 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108428.log 2024-01-27T04:27:28.917 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39324.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2024-01-27T04:27:28.929 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41362.log 2024-01-27T04:27:28.930 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.108428.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108428.log.gz 2024-01-27T04:27:28.943 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40221.log 2024-01-27T04:27:28.946 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.41362.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.41362.log.gz 2024-01-27T04:27:28.953 INFO:teuthology.orchestra.run.smithi110.stderr: 92.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38919.log 2024-01-27T04:27:28.959 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40221.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40783.log.gz 2024-01-27T04:27:28.959 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40221.log.gz 2024-01-27T04:27:28.959 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-01-27T04:27:28.961 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38919.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38919.log.gz 2024-01-27T04:27:28.961 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39684.log 2024-01-27T04:27:28.974 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40463.log 2024-01-27T04:27:28.975 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.39684.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39684.log.gz 2024-01-27T04:27:28.988 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107968.log 2024-01-27T04:27:28.998 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.40463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38923.log 2024-01-27T04:27:29.005 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.107968.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-27T04:27:29.005 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107968.log.gz 2024-01-27T04:27:29.007 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.38923.log: 87.5% 90.1% 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40463.log.gz 2024-01-27T04:27:29.007 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-01-27T04:27:29.007 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38923.log.gz 2024-01-27T04:27:29.176 INFO:teuthology.orchestra.run.smithi162.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-27T04:27:29.223 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.c.log: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.40469.log.gz 2024-01-27T04:27:29.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-01-27T04:27:30.094 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40250.log.gz 2024-01-27T04:27:31.093 INFO:teuthology.orchestra.run.smithi110.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-27T04:27:31.352 INFO:teuthology.orchestra.run.smithi162.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-27T04:27:31.730 INFO:teuthology.orchestra.run.smithi110.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-27T04:27:35.366 INFO:teuthology.orchestra.run.smithi162.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-27T04:27:35.630 INFO:teuthology.orchestra.run.smithi162.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-27T04:27:35.962 INFO:teuthology.orchestra.run.smithi110.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-27T04:27:35.976 INFO:teuthology.orchestra.run.smithi110.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-27T04:27:36.145 INFO:teuthology.orchestra.run.smithi162.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-27T04:27:36.165 INFO:teuthology.orchestra.run.smithi162.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-27T04:27:36.166 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-27T04:27:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m7.506s 2024-01-27T04:27:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:user 0m30.669s 2024-01-27T04:27:36.167 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m1.100s 2024-01-27T04:27:36.595 INFO:teuthology.orchestra.run.smithi110.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-27T04:27:36.880 INFO:teuthology.orchestra.run.smithi110.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-27T04:27:36.882 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-01-27T04:27:36.882 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m8.253s 2024-01-27T04:27:36.882 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m36.996s 2024-01-27T04:27:36.882 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m1.257s 2024-01-27T04:27:36.883 INFO:tasks.ceph:Archiving logs... 2024-01-27T04:27:36.883 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/remote/smithi110/log 2024-01-27T04:27:36.884 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-27T04:27:41.996 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/remote/smithi162/log 2024-01-27T04:27:41.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-27T04:27:46.440 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-27T04:27:46.442 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-27T04:27:46.451 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5ffe0641f18e6ec692535294010d9456630e7923/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-27T04:27:46.451 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-27T04:27:46.452 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-27T04:27:46.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-27T04:27:46.545 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-27T04:27:46.553 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-27T04:27:46.553 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T04:27:46.556 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T04:27:46.569 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-01-27T04:27:46.573 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-01-27T04:27:46.573 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T04:27:46.574 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-01-27T04:27:46.574 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +337us[ +337us] +/- 78ms 2024-01-27T04:27:46.574 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 +110us[ +113us] +/- 66ms 2024-01-27T04:27:46.574 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 -342us[ -342us] +/- 63ms 2024-01-27T04:27:46.574 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T04:27:46.578 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T04:27:46.578 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-01-27T04:27:46.578 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 +304us[ +301us] +/- 78ms 2024-01-27T04:27:46.578 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 +98us[ +95us] +/- 66ms 2024-01-27T04:27:46.578 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 8 -348us[ -351us] +/- 63ms 2024-01-27T04:27:46.578 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T04:27:46.579 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-27T04:27:46.588 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-27T04:27:46.588 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-27T04:27:46.602 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-27T04:27:46.685 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-27T04:27:46.756 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T04:27:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T04:27:46.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T04:27:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T04:27:46.819 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-27T04:27:46.828 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-27T04:27:46.845 INFO:teuthology.task.internal:Duration was 3817.743810 seconds 2024-01-27T04:27:46.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-27T04:27:46.869 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-27T04:27:46.869 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-27T04:27:46.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-27T04:27:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T04:27:46.916 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T04:27:47.223 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-27T04:27:47.223 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-01-27T04:27:47.223 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-27T04:27:47.243 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-01-27T04:27:47.243 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-27T04:27:47.269 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-27T04:27:47.269 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-27T04:27:47.285 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-27T04:27:47.339 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-27T04:27:47.339 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-27T04:27:47.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-27T04:27:47.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-27T04:27:47.488 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-27T04:27:47.488 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-27T04:27:47.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-27T04:27:47.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-27T04:27:47.554 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-27T04:27:47.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-27T04:27:47.580 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-01-27T04:27:47.607 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-01-27T04:27:47.651 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-27T04:27:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T04:27:47.666 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-27T04:27:47.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T04:27:47.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-27T04:27:47.715 INFO:teuthology.task.internal:Transferring archived files... 2024-01-27T04:27:47.716 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/remote/smithi110 2024-01-27T04:27:47.716 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-27T04:27:47.772 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300/remote/smithi162 2024-01-27T04:27:47.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-27T04:27:47.808 INFO:teuthology.task.internal:Removing archive directory... 2024-01-27T04:27:47.808 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-27T04:27:47.810 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-27T04:27:47.857 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-27T04:27:47.871 INFO:teuthology.task.internal:Not uploading archives. 2024-01-27T04:27:47.871 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-27T04:27:47.888 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-27T04:27:47.888 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-27T04:27:47.890 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-27T04:27:47.904 INFO:teuthology.orchestra.run.smithi110.stdout: 265291 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 27 04:27 /home/ubuntu/cephtest 2024-01-27T04:27:47.912 INFO:teuthology.orchestra.run.smithi162.stdout: 265291 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 27 04:27 /home/ubuntu/cephtest 2024-01-27T04:27:47.913 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-27T04:27:47.923 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-27T04:27:47.938 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-27T04:27:47.939 INFO:teuthology.nuke:Checking targets against current locks 2024-01-27T04:27:47.961 DEBUG:teuthology.nuke:shortname: smithi110 2024-01-27T04:27:47.962 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-27T04:27:47.987 DEBUG:teuthology.nuke:shortname: smithi162 2024-01-27T04:27:47.987 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-27T04:27:47.990 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-27 03:15:43.449956', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE7jbvzpRRcKWxC8MKwEp6/2Eg7/ebLoAtLx3fCHrCP4j6YgaKbk2gzjCrCnqBSa8R7LP8NJKUc2BF0mFs8jIH8='} 2024-01-27T04:27:48.014 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-01-27_03:09:32-rados-wip-read-balancer-mgr-module-distro-default-smithi/7535300', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-27 03:15:43.451957', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJpZBxOlpfYmIDdeeZrTsodR/7nDNca1GqxhvsdenHo1pug6H89ZquNR7aY5aBc19vkT9OGARRG+poLeF0c8MI4='} 2024-01-27T04:27:48.044 INFO:teuthology.orchestra.console.smithi110:Power off 2024-01-27T04:27:48.044 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-27T04:27:48.066 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2024-01-27T04:27:48.067 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T04:27:48.080 INFO:teuthology.orchestra.console.smithi162:Power off 2024-01-27T04:27:48.081 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-27T04:27:48.090 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-01-27T04:27:48.102 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-01-27T04:27:48.102 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T04:27:48.124 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-01-27T04:27:52.091 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T04:27:52.125 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T04:27:52.202 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-01-27T04:27:52.235 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-01-27T04:27:56.204 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T04:27:56.236 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T04:27:56.315 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2024-01-27T04:27:56.347 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-01-27T04:27:56.415 INFO:teuthology.orchestra.console.smithi110:Power off completed 2024-01-27T04:27:56.448 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-01-27T04:27:56.552 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3817.7438101768494 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_lflores@teuthology status: fail success: false 2024-01-27T04:27:56.552 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-27T04:27:56.628 INFO:teuthology.run:FAIL